DNS reporting data issues
Incident Report for DNSWatch (formerly Strongarm)
Update
We are continuing to monitor the processing of our DNS reporting data, but systems are stable after modifications made yesterday.

DNS resolution and other services are currently operating normally.
Posted Nov 21, 2019 - 09:25 EST
Update
Earlier today an incident occurred that caused intermittent access to the blackhole. This affected other systems and resulted in lost DNS reported data from 9:00 AM Eastern to noon Eastern.

A mitigation has been put in place to hopefully avoid additional data loss, but we are continuing to monitor these services.

DNS resolution has not been impacted by this event. Customers have continued to be protected throughout this incident.
Posted Nov 20, 2019 - 15:25 EST
Update
Intermittent access to the blackhole is continuing. Customers are still protected but may not see an education block page.

We are continuing to monitor the processing of our DNS reporting data. Some reporting data from 2:00 PM Eastern to 4:00 PM Eastern was lost.

DNS resolution has not been impacted by this event. Customers continue to be protected.
Posted Nov 19, 2019 - 15:33 EST
Update
We are continuing to monitor the processing of our DNS reporting data. Some backend services were affected causing intermittent access to the blackhole and some reporting data from 9:00 AM Eastern to 11:00 AM Eastern to be lost.

DNS resolution has not been impacted by this event. Customers have continued to be protected, but might not see an education block page in some situations. Other services are operating as expected.
Posted Nov 19, 2019 - 12:07 EST
Update
To avoid impact on additional services, some DNS reporting data was lost between 9:00 AM Eastern and 12:00 PM Eastern time for November 15th, 2019. Some blackhole alerts were delayed in processing.

We are continuing to monitor the situation and are working on improvements.

DNS resolution has not been impacted by this event. Customers have continued to be protected. The blackhole and other services are currently operational.
Posted Nov 15, 2019 - 12:08 EST
Update
To avoid impact on additional services, some DNS reporting data was lost between 3:00 AM Eastern and 7:00 PM Eastern time for November 14th, 2019. Some blackhole alerts were delayed in processing.

We are continuing to monitor the situation and are working on improvements.

DNS resolution has not been impacted by this event. Customers have continued to be protected. The blackhole and other services are currently operational.
Posted Nov 14, 2019 - 18:52 EST
Update
To avoid impact on additional services, some DNS reporting data was lost between 11:00 AM Eastern and 1:00 PM Eastern time for November 13th, 2019. Some blackhole alerts were delayed in processing.

We are continuing to monitor the situation and are working on improvements.

DNS resolution has not been impacted by this event. Customers have continued to be protected. The blackhole and other services are currently operational.
Posted Nov 13, 2019 - 12:56 EST
Update
We are investigating why the blackhole service is intermittently unavailable.

DNS resolution and other services are operating as expected. DNSWatch customers are still protected.
Posted Nov 13, 2019 - 12:05 EST
Update
Last night our monitoring alerted us to issues stemming from the DNS reporting data. From approximately 8:00 PM to 8:30 PM the blackhole was having issues responding to queries. In order to contain the impact, DNS data from 5:00 PM Eastern to 9:00 PM Eastern was lost.

We recognize this processing is currently unstable and are continuing to work toward improvements.

At no point was DNS resolution or customer protection impacted.

Currently all systems are operational. DNS reports are currently being processed, but are delayed.
Posted Nov 12, 2019 - 12:27 EST
Update
We are monitoring the processing of DNS reporting data and investigating ways to improve the efficiency and resiliency of this processing chain. DNS reports are currently delayed, but processing is ongoing.

Currently all services are operating normally.
Posted Nov 11, 2019 - 11:12 EST
Monitoring
DNS reporting processing is continuing. While troubleshooting this we realized that the impact was greater than initially realized.

DNS reporting data from approximately 10:00 PM Eastern Nov. 6th to 6:00 PM Eastern Nov. 7th was lost. Alerts during the time period of roughly 8:00 AM Eastern Nov 7th to 6:00 PM Eastern Nov. 7th are lost.

DNS resolution was not affected during this incident. Customers were fully protected, but might not have seen the educational block page. All services are operating normally.
Posted Nov 07, 2019 - 18:14 EST
Update
We've identified the ongoing issue and are working to remediate the problem. Recent reporting data (starting at roughly 10 PM Eastern time on November 6th, 2019) will not be available.

The blackhole service is currently not down, but users are still protected (they are just not being served education content).

We're investigating reports of issues logging into the DNSWatch dashboard.

DNS resolution and other components are unaffected.
Posted Nov 07, 2019 - 16:31 EST
Update
We are still investigating the current issues processing DNS reporting data. You may find that recent reporting data is not appearing in the DNSWatch dashboard.

DNS resolution and other services are operating normally.
Posted Nov 07, 2019 - 11:04 EST
Investigating
We are currently investigating issues processing of DNS reporting data. DNS resolution is not impacted and is working normally.

We are also investigating some intermittent issues with the blackhole service.

Other services, including DNS resolution, are not affected.
Posted Nov 07, 2019 - 08:31 EST
This incident affects: Dashboard and Blackhole.