Our upstream providers have provided us with a complete RFO. This service outage was caused by the same breaker at our London data centre failing as the outage on the 12th of June. Our upstream provider proceeded to change this breaker during the downtime to prevent future issues instead of waiting for the overnight maintenance window. We do not expect this issue to happen again, however we are concerned that a single breaker failure can cause widespread disruption and will be engaging with our upstream provider to ascertain how this can be alleviated.
12:48 – A service impacting issue has been detected. Our engineers are investigating as a matter of critical urgency.
12:59 – Our engineers have confirmed with our upstream providers that they are currently experiencing problems impacting all routing in and out of our primary datacentre in London.
13:43 – Service is being restored, we are proceeding to validate that all services are once more running as expected and will provide an update as soon as we are able to confirm full service restoration.
13:53 – We can confirm complete service has been restored. The cause of the outage is due to power failure at the datacentre. We believe this is directly linked to the previous incident and are chasing our upstream provider for the RFO we expect for the previous incident as well as this one. Emergency remedial work for this issue is expected to be performed this night with no service disruption.
We apologise for the inconvenience and will provide a further update once we receive a complete RFO from our upstream provider.