Datacenter outage OVH

[QuickBreakdown 1948]

Servers and forum were unreachable this morning due to a problem at OVH (where our forum & server is hosted).

 
Source: http://travaux.ovh.com/?do=details&id=28256
 

Hello,
Before all the details, two first informations.

This morning we had 2 separate incidents that have nothing to do with each other. The first incident affects our Strasbourg site (SBG) and the 2nd Roubaix (RBX). On SBG we have 3 datacentres in operation and 1 under construction. On RBX, we have 7 datacentres in operation.

SBG:
On SBG we had an electrical problem. Power has been restored and services are restarting. Some customers are UP and others not yet.
If your service is not yet UP, the recovery time is between 5 minutes and 3-4 hours. Our monitoring system allows us to know which customer is still impacted and we are working to fix them.

RBX:
We had a problem on the optical network that allows RBX to be connected with the interconnection points we have in Paris, Frankfurt, Amsterdam, London, Brussels. The origin of the problem is a software bug on the optical equipment which caused the loss of the configuration and the cut of the connection with our site of RBX. We handed over the backup of the software configuration as soon as we diagnosed the source of the problem and the DC is reachable again. The incident on RBX is closed. With the manufacturer, we are looking for the origin of the software bug and also how to avoid this kind of critical incident.

We are in the process of retrieving the details to provide you with information on the SBG recovery time of all services / clients. Also, we will give all the technical details on the origin of these 2 incidents.

We are sincerely sorry. We have just experienced 2 simultaneous and independent events that impacted all RBX customers between 8:15 am and 10:37 am and all SBG customers between 7:15 am and 11:15 am. We continue to work on clients who are not yet UP at SBG.

Regards
Octave