Database experiencing high load - Dashboard and Blackhole impacted
Incident Report for DNSWatch
Resolved
As of October 3, at approximately 1PM Eastern, all DNSWatch services returned to fully operational.

If a user visits a malicious or suspicious domain, they will see the DNSWatch block pages and an alert will be generated as expected. The Blackhole service is fully operational.

The high CPU usage on the primary database was the result of more customers enabling content filtering and visiting filtered domains, resulting in more data being recorded, combined with a set of under performing database queries.

The DNSWatch Engineering team improved the performance of many of these queries and has taken steps to provision additional database resources in order to avoid this situation in the future.

Throughout this incident, customers were still protected by DNSWatch. DNS and the DNSWatchGO Client APIs remained unaffected.

We apologize for this inconvenience and encourage any customers with questions to contact WatchGuard Support.
Posted Oct 09, 2019 - 11:46 EDT
Update
We are continuing to investigate the root cause of this issue.

Currently, if a user visits a malicious or suspicious domain, they will see the DNSWatch block pages but will not generate an alert. We have intentionally disabled alerting during this incident in order to help ensure all other services are not impacted.
Posted Oct 02, 2019 - 11:13 EDT
Update
We have taken steps to restore access to the Dashboard. The DNSWatch Dashboard is now fully operational.
Posted Oct 02, 2019 - 07:11 EDT
Update
High load on our primary database is causing a partial outage of the Blackhole service and a full outage of the DNSWatch Dashboard. Unfortunately, customers are currently unable to login to DNSWatch.

All other services, including DNS, are fully operational. Despite not being able to see the block page, customers are still protected.

We are currently investigating and apologize for this inconvenience!
Posted Oct 02, 2019 - 05:19 EDT
Investigating
We are currently investigating high load on our Blackhole service.

Customers are intermittently unable to see their customized block page when visiting a blocked domain. Customer alerts may be delayed in appearing in the dashboard.

All other services, including DNS, are fully operational. Despite not being able to see the block page, customers are still protected.
Posted Sep 24, 2019 - 12:24 EDT
This incident affected: Dashboard and Blackhole.