SWITCHlan Trouble Tickets

Transparency is very important to us. We therefore publish all trouble tickets about issues that have an impact on our customers. We show both currently open tickets as well as all closed tickets.

Ticket 20180809_1

Ticket Number: 20180809_1Ticket State: CLOSED
Ticket Opened: 2018-08-09 08:15Ticket Closed: 2018-08-24 11:12
Ticket Description: Network problems in SWITCHlan

Problem Description:

We currently observe slow network connections and packet loss in our network. Several customers report issues to reach some (web) sites.
The cause of the problem is unknown yet.

Affected:

From 2018-08-08 22:10 until 2018-08-09 11:10
Impact: Partial loss of connectivity
Service: P2P/OPN
Sites: all OPN services based on EoMPLS or VPLS (no transponder links)

From 2018-08-08 22:10 until 2018-08-09 11:10
Impact: Partial loss of connectivity
Service: network access
Sites: all SWITCHlan customers

Actions:


2018-08-24 11:12
The root cause analysis with Cisco is still ongoing. But since this issue has not re-appeared for more than 10 days now, we close this ticket. A detailed report about this incident has been sent separately to all customers.

2018-08-09 11:18
The problem could be traced down to the router swiEZ3 having problems with the forwarding of packets. Though we tried to find a root cause, the only option that resolved the issue was a complete reload of the device. Since 11:10 today, the routing in our backbone network has been stable again. There should be no more connectivity problems for any of our customers. If you still encounter problems, please contact noc@switch.ch directly. There will be a detailled explanation of this incident including lessons learned within the next few days. We also informed Cisco about this incident in order to gather further information about the root cause.

2018-08-09 10:04
Customers are still reporting connection problems (high RTT and packet loss). Investigation is ongoing.

2018-08-09 09:43
A link between two of our routers in the city of Zurich is faulty and produces packet loss. This generates routing issues in our network.
This faulty link has now been disabled. Since then we observe a stabilization of the network. Nevertheless we keep investigating.

2018-08-09 08:26
Added switch-oper@switch.ch to the list of ticket receivers.


All times are in local Swiss time CET / CEST.
For all questions about this ticket, please send mail to noc@switch.ch
or call +41 44 268 15 30.