SWITCHlan Trouble Tickets

Transparency is very important to us. We therefore publish all trouble tickets about issues that have an impact on our customers. We show both currently open tickets as well as all closed tickets.

Ticket 20080610_1

Ticket Number: 20080610_1Ticket State: CLOSED
Ticket Opened: 2008-06-10 17:55Ticket Closed: 2008-06-20 16:19
Ticket Description: Reachability issues with ETHZ

Problem Description:

Connectivity to ETHZ and some associated organizations such as EMPA was interrupted.

Affected:

From 2008-06-10 17:28 until 2008-06-10 17:52
Impact: Total loss of connectivity
Sites/Services: ETHZ


Actions:


2008-06-20 16:37
In the meantime, the problem has been analyzed as follows:
New transit connections with Global Crossing in Geneva were activated that afternoon. Due to a bad interaction (likely software bug) of the new default route with existing configuration on our router swiEZ2.switch.ch, that router stopped advertising the default route to ETHZ over BGP-4.
Traffic from the Internet to ETHZ continued to flow over the primary connection, while return traffic used the backup connection via swiZH2.switch.ch (at UZH). Inbound policy on ETHZ' access router was implemented using a "reverse path forwarding" (RPF) check. This caused incoming traffic on the primary connection to be blocked in the case of asymmetric routing.
On the SWITCH side, the configuration was changed so that the default route is reliably sent again, although there are still possible (but highly improbable) situations where asymmetric routing could occur. ETHZ is looking for ways to mitigate the impact of RPF in case such a situation reoccurs.

2008-06-19 17:00
ETHZ informed us that the ingress filter (RPF) has been configured again on their primary SWITCH connection.

2008-06-11 09:07
The primary connection between swiEZ2.switch.ch and ETHZ was reactivated. A modification of unrelated configuration on our router caused routing to become symmetric again.
As a temporary solution, ETHZ has disabled RPF (reverse path forwarding) filtering on the port, so that traffic could flow even in case routing would become asymmetric again.

2008-06-10 17:51
Under the assumption that there is an issue with ETHZ' primary router, we decided to shut down the primary BGP peering. Traffic now flows via ETHZ' secondary connection (at UZH), and indeed connectivity seems to be working again now.

2008-06-10 17:35
We noticed that services such as www.ethz.ch, www.empa.ch are no longer reachable. However, both the physical links and the routing protocols seem to work fine between SWITCH and ETHZ.


For all questions about this ticket, please send mail to noc@switch.ch
or call +41 44 268 15 30.