You’re right we have surprisingly few major network outages which is why when I hear of one it raises my suspicions...
In the case of Delta, they shutdown their worldwide network because of a power failure at a data center in Atlanta, according to Delta.
In the case of United they claimed their worldwide outage was due to a router configuration changed in Chicago...
I’m calling BS on both of those cases....
In Delta’s case if a power failure in Atlanta caused a worldwide outage, if that’s true, then a whole bunch of IT people need to be immediately fired.....
In United’s case, I’ve made enough router changes in worldwide networks to know, if that is true, again a whole bunch of IT people need to be fired....
I’m intimately familiar with the Change Control process of major companies, one of the things required to make changes is a testing procedure after changes are made and a back out procedure if your changes fail.....
I’ve seen multiple times were changes went south on a worldwide network that caused outages, none shut down the entire network and none required hours to resolve.
I saw a guy put in a router change that took down the entire east coast ATM network of a major bank, that was corrected in less than 1 hour...
I’ve seen a guy take down over 20,000 voice mail accounts from a major account that took about 3-4 hours to correct....
when this type of thing happens usually a root cause analysis is done and when all the investigation is done, people get disciplined.....
I almost was fired from a job one time when I was accuse of operating outside a change control which I did, but my excuse was I did not know the change control was input incorrectly...
I produced my documentation that clearly described the change, the issue was the person inputting the change did not include all the router names and changes....
Even though I operated outside of the change control, nothing was taken down but I was still almost fired.
I sure hope these pilots know how to land these planes in case of computer failure, or failure of the air traffic control systems or what have you. I’m sure they do, but it seems everything is possible now with these cyberattacks.
And the whole point of distributed computing (internet) was to avoid situations where a single failure would shut down the entire network. Apparently that does not get covered in the beginner courses anymore.
I buy both of those cases.. It’s right in line with my experience in the industry. Company I work for is acquisition based, so we’re all over the world. A while back they decided that key elements of our code building should be centralized to avoid... whatever I never figured that out. And technically speaking they also built redundancy into it. Problem, while they did centralize to multiple locations the build system can’t talk to more than one. When Texas had their power outage we lost our Austin server. We in Tucson couldn’t run a build for a week.
I’m intimately familiar with change control of major companies too. Familiar enough to know there’s what they SAY and what they DO. And they never do what they say.
I’ve seen equivalents to everything you say you’ve never seen. I work on enterprise level software that’s used by over 400 of the Fortune 500, the frequency with which somebody knocks us down and insists they change nothin only they did is at least once a month.