On July 29, 2020, the primary DNSWatch DNS nameserver in the Americas (USA / US-EAST) region experienced a disruption. During this time, US based partners and customers with DNSWatch enabled on their Fireboxes experienced intermittent issues connecting to the internet. We sincerely apologize for any inconvenience this caused and want to share more details about what happened, what we've already changed, and what we're working on to ensure DNSWatch provides the reliability you expect and deserve.
A full timeline of this incident, as well as other issues and incidents, is tracked on our public DNSWatch statuspage.
Following a routine deployment, our primary DNS nameserver in the Americas region stopped responding to DNS requests. As a result, US based customers and partners with DNSWatch enabled on their Fireboxes experienced intermittent issues connecting to the internet from 4:52PM UTC to 8:10PM UTC.
All customers and partners with GO Clients and protected networks deployed were not impacted during this incident; only US based customers with DNSWatch enabled on their Firebox were impacted.
The incident resulted from our primary DNS nameserver running out of available disk space. Despite monitoring available disk space and having alarms that alert the engineering team, this lack of disk space went unaddressed. When deploying to these systems, processes were unable to be restarted, and the systems failed to recover.
Additionally, our team found that during primary DNS nameserver failures, users behind a Firebox will experience internet connection issues while the device shifts to using the secondary nameserver.
The engineering team allocated more disk space to the impacted systems and restarted services, restoring DNS services.
We are taking three actions moving forward:
You can monitor the status of DNSWatch services and receive email notifications by subscribing to our DNSWatch statuspage.
We sincerely apologize for the impact to our affected customers and value the opportunity to meet your security needs.