SWITCHlan Trouble Tickets

Transparency is very important to us. We therefore publish all trouble tickets about issues that have an impact on our customers. We show both currently open tickets as well as all closed tickets.

Ticket 20080907_1

Ticket Number: 20080907_1Ticket State: CLOSED
Ticket Opened: 2008-09-07 17:47Ticket Closed: 2009-02-02 14:05
Ticket Description: Outbound route filtering issue

Problem Description:

A configuration error prevented us from propagating some route announcements to the Internet. The result was that the university of Basel and associated institutions (FHNW, FMI) were not reachable from the Internet outside the SWITCH community.

Affected:

From 2008-09-07 10:30 until 2008-09-07 16:03
Impact: Total loss of connectivity
Sites/Services: UNIBAS


Actions:


2008-09-10 11:42
We have taken some short-term measures to improve internal review of configuration changes and are discussing further longer-term improvements with the objective of reducing the possibility of similar incidents in the future.

2008-09-07 16:08
The problem in the routing policy was found and fixed, all border routers reconfigured, and route announcements to our transit providers refreshed. Traffic from the Internet to the university and adjacent sites started flowing again.

2008-09-07 15:30
SWITCH staff started diagnosing the issue.

2008-09-07 13:49
University of Basel tried to contact SWITCH about the routing problem.

2008-09-07 10:30
The maintenance at University of Basel was completed normally. But because of the error in our outbound route filters, the university's routes were no longer being announced to our transit providers, causing lack of connectivity from the Internet (except for SWITCH and its customers) to the university and sites connected to it (FHNW, FMI).

2008-09-07 10:02
The University of Basel performed maintenance on their routers with the connections to SWITCH. Both routers were upgraded separately so that connectivity would always be maintained.

2008-09-04 14:47
The file with the incorrect filters was installed on the last transit router (swiIX2). Since then, route changes for most customers would not have been propagated to our transit providers.

2008-07-23 17:25
Some comments were added to the template file for outbound BGP route filters, but in a way that broke them.


For all questions about this ticket, please send mail to noc@switch.ch
or call +41 44 268 15 30.