Ticket 20120629_1

Ticket Number: 20120629_1Ticket State: CLOSED
Ticket Opened: 2012-06-29 01:09Ticket Closed: 2012-06-29 16:07
Ticket Description: Power interruption in both PoP's in Zurich

Problem Description:

On Thursday, 28 June 2012, a short circuit in their main power switch caused a fire at ETH Zürich. When UPS batteries were drained at 20:10 and 20:33 respectively, both datacenters at the ETH Zurich suffered a complete power failure. This had significant impact on SWITCH's network and services.
Full network connectivity was restored at around 23:00, but recovery of most services could only begin once the building CLA was powered up as well. All SWITCH services were available again at 02:30 on Friday, 29 June 2012.
A few pieces of equipment did not survive the incident and will be replaced as soon as possible.

Affected:

From 2012-06-28 20:10 until 2012-06-28 22:57
Impact: Total loss of connectivity
Service: network access
Sites: ITIS

From 2012-06-28 20:10 until 2012-06-28 22:57
Impact: no more redundancy
Service: network access
Sites: CSCS, ENSI, ETH, WSL


From 2012-06-28 20:10 until 2012-06-28 22:57
Impact: Total loss of connectivity
Service: P2P/OPN
Sites: CNAF-GRIDKA-LHCOPN-001


From 2012-06-28 20:10 until 2012-06-28 22:57
Impact: no more redundancy
Service: P2P/OPN
Sites: ETHZ-BSSE-ez2-ba2, ETHZ-CSCS-ez1-lug1, ETHZ-ETHBOARD-ez2-be2


From 2012-06-28 20:10 until 2012-06-28 22:57
Impact: Total loss of connectivity
Service: network access
Sites: SWITCH

From 2012-06-28 20:10 until 2012-06-28 23:00
Impact: Total loss of connectivity
Service: network access
Sites: NAZ


Actions:


2012-06-29 00:38
The CLA datacenter is back online. Most services are restored. We're still working to get everything up and running.

2012-06-28 22:57
The RZ datacenter is back online. CLA still down.

For all questions about this ticket, please send mail to noc@switch.ch
or call +41 44 268 15 30.