SWITCHlan Trouble Tickets

Transparency is very important to us. We therefore publish all trouble tickets about issues that have an impact on our customers. We show both currently open tickets as well as all closed tickets.

Ticket 20171115_1

Ticket Number: 20171115_1Ticket State: CLOSED
Ticket Opened: 2017-11-15 16:19Ticket Closed: 2017-11-23 10:08
Ticket Description: P2P/OPN (Layer-2) problem

Problem Description:

We got several complaints (from several customers), that their P2P/OPN links within SWITCHlan are broken. The problem could not be located yet. We suspect an issue with MPLS on one of our main routers.


From 2017-11-15 15:45 until 2017-11-15 16:20
Impact: Partial loss of connectivity
Service: P2P/OPN
Sites: CERN-CNAF-LHCOPN-002, EHB-LG, EHB-LS, EHB-ZO, EmpaEawag-KAS, EmpaEawag-SGE, EmpaEawag-THU, ENSI-NAZ, EPFL-CBT-el2-ge2, EPFL-CBT-ls2-ce2, EPFL-CSCS-ce2-lug1, EPFL-IMT-el2-ne2, EPFL-IMT-ls2-ne1, EPFL-LABA-el2-ba1, EPFL-Lausanne, EPFL-Sion, ETHZ-BSSE-ez3-ba2, ETHZ-BSSE-zh1-ba1, ETHZ-CSCS-ez3-lug1, ETHZ-CSCS-zh1-lug2, ETHZ-ETHBOARD-ez2-be1, FFHSRD-FFHSBR, HEARCNE-HEARCDEL-ne1-del2, HESSO-AVP, HESSO-DEL, HESSO-ECAL, HESSO-EESP, HESSO-EIC, HESSO-HEFR, HESSO-HEIGVD, HESSO-HESAV, METEOCH, METEOCH-CSCS, NTB-HTWCHUR-bu2-cr2, NTB-Waldau-bu2-fhsg2, RERO-UNIL-my2-ls2, Sommet-BLU, Sommet-BUL, Sommet-CL, Sommet-GL, UNIBE-HFSJGJFJ-be2-jfj2, UNIBE-UNIFR-be2-fr1, UNIL-IUKB-ls2-si2, VSnet-vi2-my2, WSL-CAD-wsl2-caw2, WSL-SLF-wsl1-slf2, WSL-SLF-wsl2-slf1


2017-11-23 10:09
The root cause of this outage turned out to be a memory leak on our backbone router swiCE1 at CERN. During announced maintenance work at EPFL, the physical link between swiCE1 at CERN and swiEL1 at
EPFL had to be disrupted for a short moment. For as long as the physical link was disrupted, all traffic was routed via another path. Once the link came up again, the router swiCE1 did start the OSPF process.
However, the LDP process was not started due to an out-of-memory condition. The effect was that OSPF signalled the link between swiCE1 at CERN and swiEL1 at EPFL as active while the forwarding of MPLS traffic didn't take place anymore due to the missing label distribution via LDP. This led to a blackholing situation for all OPN traffic passing this link.

2017-11-20 22:00
The firmware of swiCE1 has been upgraded. All services are restored. Closing ticket.

2017-11-15 16:51
It seems like the problem has been solved since approx. 16:25.
All links are back to normal operation. Please contact us (noc@switch.ch) if you observe any anomalies.

For all questions about this ticket, please send mail to noc@switch.ch
or call +41 44 268 15 30.