SWITCHlan Trouble Tickets

Transparency is very important to us. We therefore publish all trouble tickets about issues that have an impact on our customers. We show both currently open tickets as well as all closed tickets.

Ticket 20070509_2

Ticket Number: 20070509_2Ticket State: CLOSED
Ticket Opened: 2007-05-09 17:57Ticket Closed: 2007-05-09 22:46
Ticket Description: Routing problem for PSI after configuration of redundant access

Problem Description:

In the process of reconfiguring PSI's SWITCH access to a fully redundant setup, two small configuration mistakes were made that caused routing problems for traffic from some parts of the Internet towards parts of PSI's network.

Affected:

From 2007-5-8 21:20 until 2007-5-9 09:38
Impact: Partial loss of connectivity
Sites/Services: PSI


Actions:


2007-05-09 10:56
The interior routing problem for PSI's secondary IP address range (192.33.120.0/21) in SWITCH's backbone was fixed. BGP advertisements to the Internet became stable again. Parts of the Internet continued to suppress the route ("dampening") for up to forty minutes.

2007-05-09 09:25
The MPLS configuration problem was fixed.

2007-05-08 21:20
Finalization of configuration on swiPS1, swiPS2, and external (BGP) border routers. Two configuration errors were apparently introduced at this stage. These caused two separate problems:
(1.) MPLS forwarding was accidently configured for PSI's address range. Normally, MPLS forwarding is NOT used to forward "normal" IP packets on SWITCH's backbone. For reasons that we do not completely understand, MPLS forwarding towards PSI didn't work reliably on all routers in our backbone.
(2.) The route for PSI's secondary IP address aggregate became unstable within SWITCH's interior routing. This instability was propagated to the announcements of that route towards the general Internet. The result was that, at first, connectivity from most of the Internet to this PSI address range became intermittent. In addition, some parts of the Internet started to suppress routing to the instable prefix altogether ("route dampening"). Since this secondary prefix contains all authoritative and recursive DNS servers for PSI, both access from PSI to the Internet and access from the Internet to PSI resources (such as Web sites) was severely disturbed.

2007-05-08 16:30
Activation of redundant access of PSI. PSI is now connected to two new SWITCH routers, swiPS1 and swiPS2.


For all questions about this ticket, please send mail to noc@switch.ch
or call +41 44 268 15 30.