Outages following Nashville bombing spotlight vulnerable U.S. voice, data networks

KTLA ^ | 12/31/2020

[BenLurkin]

The blast seriously damaged a key AT&T network facility, an important hub that provides local wireless, internet and video service and connects to regional networks. Backup generators went down, which took service out hours after the blast. A fire broke out and forced an evacuation. The building flooded, with more than three feet of water later pumped out of the basement; AT&T said there was still water on the second floor as of Monday.

The immediate repercussions were surprisingly widespread. AT&T customers lost service — phones, internet or video — across large parts of Tennessee, Kentucky and Alabama. There were 911 centers in the region that couldn’t take calls; others didn’t receive crucial data associated with callers, such as their locations. The Nashville police department’s phones and internet failed. Stores went cash-only.[*****]

At some hospitals, electronic medical records, internet service or phones stopped working. The Nashville airport halted flights for about three hours on Christmas. Rival carrier T-Mobile also had service issues as far away as Atlanta, 250 miles away, because the company uses AT&T equipment for moving customer data from towers to the T-Mobile network.

...

Natural disasters like hurricanes frequently wipe out service as the power goes out and wind, water or fire damage infrastructure. Recovery can take days, weeks or even longer. Hurricane Maria left Puerto Rico in a near communications blackout with destroyed telephone poles, cell towers and power lines. Six months later there were still areas without service.

Software bugs and equipment failures have also caused widespread problems. A December 2018 CenturyLink outage lasted for more than a day and disrupted 911 calls in over two dozen states and affected as many as 22 million people.

[BenLurkin]

Which goes to show that somebody can be a kook, and still be clever.

[montag813]

[COBOL2Java]

Disaster recovery is a crucial component of software engineering.

[Bell Bouy II]

ATT survived WTC attack on 9/11 although for obvious reasons cell service was taken out. Which is one reason I keep my POTS
Short clip below

I worked in S Fl CO’s with some of these Western peeps that later transferred to Broward in the late 70’s

https://techchannel.att.com/playvideo/2011/09/09/AT&T-Archives-The-Twins

[ZOOKER]

It's an inconvenience, but not one worth wasting your life over.

And the public warnings to evacuate, what were those about?

This makes no sense at all - either the guy's totally whack or something's being left out here.

[.44 Special]

Disaster recovery is a crucial component of business continuity. Untested disaster recovery plans are a waste of paper they’re printed on. Ninety-four percent of all disaster recover plans are untested. Another four percent are partially tested. Two percent are tested. Less than 1/2 of one percent of disaster revovery plan have chance of working. Most disaster recovery plans are written without a full risk assessment being performed. A risk assessment, like disaster recovery, is not just about IT. It is about your business, every aspect of it. What’s your business worth?

[COBOL2Java]

When I worked in Disaster Recovery for Blue Cross and Blue Shield of the National Capital Area, we tested every six months. The backup tapes were kept in a cave in Pennsylvania, and we worked for 3 days straight in a control center in Fairfax, VA. Ate, slept and worked there until we were able to bring the entire system up and running.

[Bernard]

A plan is not a plan until it is tested. And the point of a DR or BC test is not to "pass" - it's to see what the decision makers available during the test do to work around "when stuff happens". Because most plans don't work.

I was observing a full mainframe DR for a large Chicago bank and the first problem was that the remote back-up storage contractor lost tapes 1, 2, and 3. That's pretty much the part of the recovery that tells the computer that it's a computer. The solution? Call the data center and have them cut tapes 1, 2 and 3 and send them to the recovery site.

Sometimes the audit reports just write themselves.

[Ruy Dias de Bivar]

I still remember, quite a few years back, the outages caused when someone got into a Tulsa telephone underground cubicle and went after the wires with a chainsaw.

[wetgundog]

Most snowsflakes would commit suicide if they were without their cellphones for four days, maybe even four hours.

[.44 Special]

Had to be careful about how far to take the explanation or you just lose people. My belief is that contracted recovery centers are probably the best bet if practical for your scenario. Main objective is to keep management out of ITs hair and let them get on with recovery. People/organizations think it will never happen to them. Best to you in the New Year.