SWITCHlan Trouble Tickets

Transparency is very important to us. We therefore publish all trouble tickets about issues that have an impact on our customers. We show both currently open tickets as well as all closed tickets.

Ticket 20200221_2

Ticket Number: 20200221_2Ticket State: CLOSED
Ticket Opened: 2020-02-21 15:50Ticket Closed: 2020-03-02 02:56
Ticket Description: Transatlantic circuit L2VPN-RKAU-SANTACLARA down

Problem Description:

On Feb 20, 17:24 CET, the transatlantic circuit L2VPN-RKAU-SANTACLARA went down. It turns out, that this problem is related to a pre-announced maintenance on GEANT's LON-WIX-ANA-100G link. The maintenance window is a full five days from 19.2 to 24.2 00:00, though only occasional outages had been expected.

If not mentioned otherwise, all times are local Swiss time, currenly CET.

Affected:

From 2020-02-20 17:24 until undefined
Impact: Total loss of connectivity
Service: P2P/OPN
Sites: L2VPN-RKAU-SANTACLARA


Actions:


2020-03-02 02:56
According to Internet2, the emergency maintenance has been completed as of 2020-02-28 12:21:33 UTC.
We're seeing live traffic since around that time and thus closing this ticket.

2020-02-27 02:17
GEANT OC has been informed that maintenance has been extended until at least Feb 29 0:00 UTC, depending on weather conditions it could be even longer.

2020-02-25 05:42
Update from Internet2 NOC:
"There appear to be lingering issues from Tata's Transatlantic maintenance last week (Feb 19 - Feb 24). All vlans should be restored now but Tata is reporting that there would be intermittent outages until Feb 26, to resolve further issues."

"VLAN 90 was unavailable due to maintenance (CHG0041106) on the DC-London transatlantic circuit, performed by the circuit vendor.
Per vendor notes, the maintenance was warranted to rectify the degrading nature of cable and restore to optimal level of operation."


2020-02-24 15:00
GEANT OC has been informed of a unannounced ongoing emergency maintenance within the circuit providers network. This has been detailed to conclude 26/02/2020 17:00 CET. Services are up however services remain at risk until this time.


2020-02-24 11:18
There seems to be traffic flowing again on the link since around 11:00 local time, however, we have not received any confirmation that this is stable. Still waiting for RFO and completion notice of maintenance.

2020-02-24 08:38
Maintenance window has passed, was supposed to end February 24 0:00 UTC.
The link still does not seem to be up again though, as we see no traffic.
We have requested more information from GEANT NOC.


2020-02-24 04:05
Internet2 is investigating with Tata (circuit provider).
No response, RFO or ETR yet.

2020-02-21 15:30
GEANT NOC points out a (pre-announced) maintenance on the affected circuit.

2020-02-21 11:10
proper operation of circuits within our network confirmed

2020-02-21 09:14
customer informs about outage

2020-02-20 17:24
L2VPN-RKAU-SANTACLARA stops carrying traffic


All times are in local Swiss time CET / CEST.
For all questions about this ticket, please send mail to noc@switch.ch
or call +41 44 268 15 30.