SWITCHlan Trouble Tickets

Transparency is very important to us. We therefore publish all trouble tickets about issues that have an impact on our customers. We show both currently open tickets as well as all closed tickets.

Ticket 20191101_2

Ticket Number: 20191101_2Ticket State: OPEN
Ticket Opened: 2019-11-01 07:10Ticket Closed:
Ticket Description: Transatlantic circuit LON-WIX-ANA-100G LL down

Problem Description:

There has been a fiber break on the 100G ANA transatlantic circuit from Washington DC (WIX) to London affecting service ID L2VPN-RKAU-INDIANAPOLIS and accordingly Link IDs:

L2VPN-RKAU-INDIANAPOLIS
L2VPN-RMUE-INDIANAPOLIS

As per concerned NOC, OTDR testing has been performed and confirmed a subsea
fiber cut about 2700KMS from the New Jersey landing station towards the UK.
Preparations are underway to align a cable repair vessel, however this is
potentially a long duration event spanning 2-3 weeks for complete repair.

If not mentioned otherwise, all times are local Swiss time, currenly CET.

Affected:

From 2019-10-31 04:02 until 2019-11-05 13:00
Impact: Total loss of connectivity
Service: P2P/OPN
Sites: RKAU-INDIANAPOLIS


Actions:


2019-11-25 04:43
The LON-WIX-ANA-100G circuit was monitored as down from 31/10/2019 03:02 UTC to
21/11/2019 23:40 UTC. Services are now up and stable GEANT have been informed
that the outage was due to damaged fibre cable, between the US towards UK.
This has now been repaired and no further outages on this circuit are anticipated.
The traffic currently re-routed via NYC-LON NEAAR circuit will be moved back to LON-WIX at a later date TBA.

2019-11-22 08:07
GEANT OC has observed the LON-WIX-ANA-100G circuit up and carrying traffic as of
21/11/2019 11:15:37UTC, GEANT is monitoring for stability and awaiting full
confirmation from concerned NOC of a stable fix.

2019-11-21 17:13
The cable ship 'Decisive' has completed
repairs to the damaged fiber. Service has unfortunately not restored and
vendor engineers are now performing loop testing to isolate a additional
potential card failure. Further updates shall be provided once known.

2019-11-19 08:57
As per concerned NOC, the issue on LON-WIX-ANA-100G is due to a damaged fiber in the sub sea section cable between the US towards UK. The cable ship 'Decisive' has arrived and has started repairs. The UK end of the fiber has been recovered and repaired. The US end of the fiber is in the process of being recovered. An ETR of November 26th is expected for restoration assuming weather conditions remain favorable. Updates will be provided when received. When repair is complete, a new date will be set to restitch the current backup circuit (via NEAAR) to LON-WIX again.

2019-11-05 13:44
GEANT has fixed a configuration issue, NEAAR attach circuit back up at around 13:00 CET.

2019-11-05 09:52
GEANT reports failure on NEAAR attach circuit in London since 07:07:40 UTC

2019-11-04 20:17
The link has been temporarily re-routed from WIX to MANLAN (NYC) to London on the NEAAR link.
Traffic is flowing again.

2019-11-01 07:15
This is a continuation of ticket SWITCH-TT 20191016_1, as that was originally tracking a scheduled maintenance.


For all questions about this ticket, please send mail to noc@switch.ch
or call +41 44 268 15 30.