The engineering team have fully investigated and identified the problem as an issue with one of our switch stacks recovering from hardware failure incorrectly. While the stack was recovering from failure one of the operational switches in the stack rolled back to an old configuration snapshot which caused connectivity to be impacted within the network, and prevent services from routing properly to internal services. Forcing the correct configuration and power cycling the hardware was enough to put it into a stable state and restore connectivity. The team have made changes to the stack's configuration to help ensure that, even in the case of unintended configuration rollback that the system will continue operating. They're also working with our hosting provider and vendors to understand and implement any further recommended remedial actions. Please accept our apologies for any inconvenience caused by this interruption in service.
All services are now back at full capacity, queues are clear and systems are working as they should be.
Posted Oct 26, 2020 - 17:27 GMT
Monitoring
The engineering team are restoring connectivity now. Most services are back up and running and currently clearing their queues.
Posted Oct 26, 2020 - 15:01 GMT
Identified
The engineering team have identified a problem in our core network at our Reading datacentre and are currently working with our hosting provider to restore connectivity.
Posted Oct 26, 2020 - 14:37 GMT
Investigating
Our engineers are currently investigating issues connecting to our application, we'll post updates as and when they are available.
Posted Oct 26, 2020 - 14:23 GMT
This incident affected: Platform Interface, Clicks, reads, unsubs, etc, API, Launches, Hosted Images and Files, SFTP Servers, Content and Spam Check, and Support.