SWITCHlan Trouble Tickets

Transparency is very important to us. We therefore publish all trouble tickets about issues that have an impact on our customers. We show both currently open tickets as well as all closed tickets.

Ticket 20070530_1

Ticket Number: 20070530_1Ticket State: CLOSED
Ticket Opened: 2007-05-30 06:36Ticket Closed: 2007-06-17 18:38
Ticket Description: Router crash at CERN (swiCE2.switch.ch)

Problem Description:

Heavy scanning traffic from a compromised machine at an university caused an overload of multicast signaling and eventually the crash of our router swiCE2 at CERN.

Affected:

From 2007-05-30 04:28 until 2007-05-30 06:12
Impact: Partial loss of connectivity
Sites/Services:
From 2007-05-30 04:28 until 2007-05-30 06:12
Impact: Partial loss of connectivity
Sites/Services:

The primary connections to GEANT2 and CERN, as well as the CIXP exchange point with some commercial peerings, were down. Traffic was flowing over alternative links, partly with lower capacity (2.5 Gb/s for the GEANT2 backup vs. 10 Gb/s for the primary connection).

From 2007-05-30 04:28 until 2007-05-30 06:12
Impact: no more redundancy
Sites/Services: CERN

Actions:


2007-05-30 10:00
Debugging and correlation with information from our security systems revealed the following: A compromised host at one of the universities connected to SWITCH started to scan the Internet at an aggressive pace. The scanning also covered the multicast space (224.0.0.0/4). This caused heavy multicast signaling traffic. Our router swiCE2 at CERN serves as a PIM-SM Rendezvous Point and as an MSDP server. It quickly became overloaded because of the scanning traffic, and eventually crashed, presumably due to a memory leak.

2007-05-30 06:05
Rebooted router manually, everything came up normally.

2007-05-30 04:48
The router stopped execution of the operating system and fell back into the boot code.

2007-05-30 04:09
Our router swiCE7:082.switch.ch started to respond slowly to control-plane requests, and some routing protocol adjacencies were lost.


For all questions about this ticket, please send mail to noc@switch.ch
or call +41 44 268 15 30.