Backbone link down, no downtime

[andy]

At 2PM Pacific Daylight time (GMT -7:00) today (Oct 29, 2018), our 1000Mbps link to CenturyLink went down.

Initial troubleshooting quickly isolated the problem to be within CenturyLink's network, somewhere between our local connection (which is up) and where we get on their layer 3 network in Spokane.

I've opened a trouble ticket with CenturyLink.

They were able to confirm that they were able to see their end of the link as up and our end of the link as down, further narrowing the problem to a layer 2 problem between Spokane and somewhere outside our facility.

I've admin'ed down our BGP session for this link to avoid any route flapping that might occur while they are doing their testing.

Once they've completed their testing and repairs, we'll resume routing on our link to CenturyLink.

Until then, traffic is routed through our secondary provider.

[andy]

We haven't received any communication from CenturyLink about this issue since opening the ticket, but we're now able to ping the remote end over the link successfully.

We're awaiting confirmation from CenturyLink that they have completed testing before we resume routing across this link.

[andy]

We still hadn't had any communication from CenturyLink regarding this ticket, so I called in to request further information.

They were able to tell me that the issue was a "failed" router card that they were able to remedy by reseating the card. They indicated that no further work was scheduled for the issue at this time.

We've resumed routing over this link.