SWITCHlan Trouble Tickets

Transparency is very important to us. We therefore publish all trouble tickets about issues that have an impact on our customers. We show both currently open tickets as well as all closed tickets.

Ticket 20071112_1

Ticket Number: 20071112_1Ticket State: CLOSED
Ticket Opened: 2007-11-12 14:59Ticket Closed: 2008-09-22 15:15
Ticket Description: Scheduled Hardwareupgrade EZ2, Thursday, 15.11.2007
Scheduled Starttime: Scheduled Endtime:

Problem Description:

We will upgrade our router swiEZ2 on Thursday, 15.11.2007 between 5:00 and 6:00.
This will result in an interruption of all connections to swiEZ2.
The time of the interruption during the upgrade is estimated at 30 minutes, but could take longer if problems occur.

The following entities are affected and have NO backup connection available:
(all traffic will be interrupted for the time of the upgrade)

- EZ2
- EZC1
- EZS1
- ETHZ Security access (UTP 41441)
- connection to NAZC1 (no backup available)
this affects
- Internet connection to NAZ
- NAZ-BIT-JuraKOMBV access (26ETZ-C1) BIT
- IPv6 tunnel uplink Verein (no backup available)
- IPv6 tunnel Kanti Zug (no backup available)
- SWITCHcam IPv6 (no backup available)
- FOXTROT
- PASCAL
- GN2 JRA1 HADES
- EZMP3

The following connections are affected, but have a backup path available:

- primary access to ETH Z├╝rich (backup via ZH2 available)
- primary EoMPLS ETH connection to Basel (backup via ZH2 available)
- primary access to SWITCH datacenter (backup via ZH2 available)
- several backbone connections to ZH2, LS2, IX1,... (backup available)
- Cablecom gateway ZUSW1 (backup available via BESW1)
- secondary access to MeteoSwiss (primary via ZH2 available)
- access to PFT (backup via ZH2 available)
- connection to HO2 (backup via MA2 available)
- connection to WI2 (backup via WI1 available)

Connections with backup paths will be re-routed accordingly.

Affected:

From 2007-11-15 05:00 until 2007-11-15 06:00
Impact: Total loss of connectivity
Sites/Services: NAZ


From 2007-11-15 05:00 until 2007-11-15 06:00
Impact: no more redundancy
Sites/Services: MCH


From 2007-11-15 05:00 until 2007-11-15 06:00
Impact: no more redundancy
Sites/Services: SWITCH


From 2007-11-15 05:00 until 2007-11-15 06:00
Impact: no more redundancy
Sites/Services: ETHZ


Actions:


2007-11-15 12:26
There was a problem with our outgoing prefix-lists and the new IOS version. Were able to fix it by adjusting a filter list entry.
The primary BGP session is now back up and everything seems to work fine.

2007-11-15 12:00
ETHZ removed RPF filters for the time being.
It seems they don't see a default route from us on the primary BPP session, so we route traffic to them via the primary session, but their traffic is flowing to us via the secondary session.

2007-11-15 08:00
Call from ETHZ to discuss options. To not disturb their users too much, decided to retry activating the primary BGP session later on at 12:00.


2007-11-15 07:00
Mail to ETHZ to inform them about the problem.


2007-11-15 06:20
Traffic to ETHZ via the primary BGP peering is not flowing properly, thus peering is shut down.
Traffic is flowing over the secondary BGP peering again.

2007-11-15 06:02
10GE XENPAK for the link to BA2 is exchanged with a new one, link comes up.

2007-11-15 05:42
New router is online and most connections are working already. 10GE connection to BA2 remains down, reason is being investigated.

2007-11-15 05:07
Router is powered off and all linecards and cables are moved to the new chassis



For all questions about this ticket, please send mail to noc@switch.ch
or call +41 44 268 15 30.