|Ticket Number:||20070509_2||Ticket State:||CLOSED|
|Ticket Opened:||2007-05-09 17:57||Ticket Closed:||2007-05-09 22:46|
|Ticket Description:||Routing problem for PSI after configuration of redundant access|
Problem Description:In the process of reconfiguring PSI's SWITCH access to a fully redundant setup, two small configuration mistakes were made that caused routing problems for traffic from some parts of the Internet towards parts of PSI's network.
Affected:From 2007-5-8 21:20 until 2007-5-9 09:38
Impact: Partial loss of connectivity
The interior routing problem for PSI's secondary IP address range (126.96.36.199/21) in SWITCH's backbone was fixed. BGP advertisements to the Internet became stable again. Parts of the Internet continued to suppress the route ("dampening") for up to forty minutes.
The MPLS configuration problem was fixed.
Finalization of configuration on swiPS1, swiPS2, and external (BGP) border routers. Two configuration errors were apparently introduced at this stage. These caused two separate problems:
(1.) MPLS forwarding was accidently configured for PSI's address range. Normally, MPLS forwarding is NOT used to forward "normal" IP packets on SWITCH's backbone. For reasons that we do not completely understand, MPLS forwarding towards PSI didn't work reliably on all routers in our backbone.
(2.) The route for PSI's secondary IP address aggregate became unstable within SWITCH's interior routing. This instability was propagated to the announcements of that route towards the general Internet. The result was that, at first, connectivity from most of the Internet to this PSI address range became intermittent. In addition, some parts of the Internet started to suppress routing to the instable prefix altogether ("route dampening"). Since this secondary prefix contains all authoritative and recursive DNS servers for PSI, both access from PSI to the Internet and access from the Internet to PSI resources (such as Web sites) was severely disturbed.
Activation of redundant access of PSI. PSI is now connected to two new SWITCH routers, swiPS1 and swiPS2.
For all questions about this ticket, please send mail to firstname.lastname@example.org
or call +41 44 268 15 30.