WebBlocker Incident Report
Users of the WebBlocker service in Europe experienced an outage late Thursday night that lasted into Friday morning, January 25 – 26, 2018. WatchGuard has worked closely with our partner Forcepoint over the last few days to analyze the failure and to put processes in place to ensure that events like this do not happen again. We are sharing details here so our partners and users are confident that we have addressed this issue.

Background
WebBlocker uses the Forcepoint ThreatSeeker Cloud URL database for web categorization, which is hosted in their ThreatSeeker Cloud Service. The URL database is hosted at 5 different locations around the world. The Firebox selects the appropriate location of the service based on the location of the DNS server that it uses. Unfortunately, there was an outage at the UK server last week that affected HTTPS lookups and led to our service outage. With Fireware version 12.0, WatchGuard switched to using the more secure HTTPS instead of HTTP for web category lookups, so only customers running Fireware version 12.0 or later were affected. Customers all over Europe use the UK server.

Incident Summary & Root Cause Analysis

• Incident start time: Thursday January 25 2018 20:49 UTC
• Incident end time: Friday January 26 2018 08:35 UTC.
• Root Cause: As part of routine maintenance of firewall infrastructure in Heathrow (A) the active Virtual IP for the ThreatSeeker Cloud service was moved to another firewall device. During this process, the firewall for the HTTPS ThreatSeeker Cloud service did not start correctly on the new device. As a result, the ThreatSeeker cloud server in the UK was not accepting HTTPS lookups, causing our service to fail. The unavailability of the HTTPS ThreatSeeker Cloud service in Heathrow (A) was not immediately detected by WatchGuard. Sufficient monitoring was not in place to check for responses to both HTTP and HTTPS requests.
• Customer Impact: Users who have the server timeout in WebBlocker configured to deny access would have lost internet connectivity during this period. Users with the alternative “fail open” setting would have seen web connections allowed but no categorization would have been provided.
• Incident Tracking: Fireboxes were unable to connect to Heathrow London aka UK (A) ThreatSeeker Cloud service using HTTPS. The incident is tracked on the Forcepoint Cloud status page at https://status.forcepoint.net/ in the ThreatSeeker Cloud section.

Process Updates
Forcepoint has increased monitoring from both HTTP and HTTPS connections to all ThreatSeeker servers around the world.  WatchGuard is also planning to put more monitoring in place to supplement the Forcepoint efforts.  WatchGuard and Forcepoint have reviewed our support escalation procedures and initiated a process to immediately elevate critical network impacting issues so they get immediate attention.

The new and enhanced monitoring, combined with more streamlined support processes, will ensure this type of incident does not occur again, as well as better and faster escalations should any future issues occur.

On behalf of WatchGuard, we apologise for any inconvenience this has caused our partners and customers.