Inside chaos at Facebook as servers crash and staff left paralysed on 'snow day'

The Mirror UK ^ | 05 October 2021 | ByMatthew Dresch

[blueplum]

Facebook staff said they experienced "what felt like a snow day" when the site's global outage left them paralysed on Monday.

...A Facebook employee reportedly said workers were not able to enter buildings to examine the scale of the outage on Monday....

....Meanwhile other members of staff at Facebook said it felt like they were having a 'snow day' as the firm's internal tools and communications also went down yesterday.

This meant no one was able to do any work, tech reporter Ryan Mac tweeted...

The issues were caused by a faulty configuration change, according to Facebook.

Explaining the outage, Facebook said: "Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centres caused issues...

[blueplum]

longer explanation at link

[Jonty30]

I presumed the employees were locked out to keep them from being witnesses to whatever was going on.

[frogjerk]

So they have a BGP issue where routes were removed from the Internet. OK. How does that prevent people from getting into the buildings? Are you telling me facebook username/password authentication at the desktop and website level was a single sign-on for every aspect of the company?

[Rebelbase]

Yep. The day before the FB whistle-blower was testifying before congress.

[blueplum]

trust the cloud?

(seriously tho, all this is over my head)

[lightman]

Enough time to wipe some server drives, etc, so that whatever the whistleblower claimed allegedly never happened.

I smell big time tampering with evidence and obstruction of justice!

[tophat9000]

So they were advertising bad routes via bgp to the rest of the world.. but bgp is an external routing protocol..

that should not impact there internal routing protocol (ospf likely) that should be handled the local keycards access.... even if they were running ibgp

Got to be some dumb ass network design they have there

[DoughtyOne]

I think it’s hilarious that Facebook got kicked off its own
platform.

Hell, they’ve enjoyed kicking so many others off. Love it!

[Army Air Corps]

Ping.

[TChad]

Hackers Warn That If Demands Aren’t Met They Will Reactivate Facebook

[frogjerk]

Management traffic should not ride the same network routing as those providing services internal or external.

Keycard access, terminal server access, and other management protocols should be independent of that type of traffic as it can affect or be affected by bandwidth and blackhole route issues.

[tophat9000]

I ran a theoretical hack scenario back in 2000 and my attack plan was to Sever all the external management connections 1st and then collapse the internal network

Something makes me think something else went on as a network collapses that bad shouldn’t happen

Interesting in my scenario the motivation was to make money by shorting a company stock before the attack so went the company went dark for a day the stock would be driven down any you could clean up on the short postion

Considering Facebook was getting hammered on capital hill and then this crash the same day

I’d be curious if there was any major shorts of their stock in the days leading up to this. Because they did take a stock hit yesterday

[brianl703]

It’s been a while since I worked on them, but the keycard access systems I dealt with stored a local copy of the access card database, so a network outage wouldn’t disable them.

[tophat9000]

Absolutely correct.. how the hell should bad bgp route Advertise to the outside world effect the internal network devices taking to each other? No way in hell that should impact the local keycards access unless a moron designated the network

If this could lock them out of the building then a wan outage of any kind that would do the same

[upchuck]

This is the Farcebook Is Evil ping list.

h/t pookie18's cartoons

Facebook is a perfect example of socialism:
You get it for free but the quality sucks.
You have no say in how it works.
The guy who runs it gets rich.
There's no real competition.
You have no privacy.
And if you say one thing they don't like
they'll shut you up.

If you'd like to be on or off this list, please click Private Reply below and drop me a FReepmail

[jdsteel]

“ Enough time to wipe some server drives, etc, so that whatever the whistleblower claimed allegedly never happened.

I smell big time tampering with evidence and obstruction of justice!”

I don’t believe in coincidences either.

[lightman]

If we had an FBI instead of a FIB there might be a half a55ed investigation.

[lulu16]

Well it worked when Trump was winning. At least they weren’t coving the windows with pizza boxes.

[\/\/ayne]

I’m not buying any of this crap! There’s some reason for internet nodes going down but I don’t believe what Facebook says.

[Islasorna]

If this was a config issue, heads are gonna ROLL at the NOC. Probably tons out the door already. IT people are the first to gone if live production goes down everytime.