[rarestia]

This outage, as well as the subsequent outage with Azure, had nothing to do with where your data lives. This could happen on your own home network as well.

DNS is basically an Internet phone book. Everything on the Internet runs using IP addresses. IP addresses are numerical and not exactly easy to remember for a large number of endpoints, so it’s not like when you were a kid and could remember your home phone number and the numbers of all of your friends. These are longer numbers and due to the sheer number of them (over 4B possible IPv4 addresses alone), a canonical system for organization makes more sense, hence DNS.

DNS works great... until it doesn’t. Over the years we’ve bolted on functionality for DNS such as more recent DNS-over-HTTPS and DNSSEC. As DNS becomes more complex, and the interoperability (recursion) of DNS systems grows, one small blip in the chain causes a massive outage downstream, and we get what we had with AWS and Azure.

The biggest problem, IMO, is that those of us who know DNS very deep and can unravel the Gordian knot it’s become are getting older, and the very fundamental functionality of the Internet as we know is becoming more anachronistic and is less of a concern or point of understanding for upcoming IT engineers. Modern communication and systems engineers aren’t taught the fundamentals in lieu of learning things like nebulous “cloud” technologies. As a result, one small change to a high level system has downstream impacts that even the most ardent UA tester can’t foresee. As those of us who’ve been doing this work for 30+ years start to eye retirement, the next generation is wholly unprepared for comprehending what was left behind, because, well, let’s face it: we were really bad about documenting it.

The Internet is going to become something akin to the Apollo program but much more impactful. All of the brains behind the origins will die off leaving a system where none of the new blood understand how the subordinate components work, and when something big breaks, we’ll likely see a massive outage that may never fully resolve.

[trebb]

I’m thinking there may be a glitch in using a combination of IPv4 and IPv6 - which was instituted because, with almost every electronic device in the world having its own IP address these days, we were running out of them under IPv4.

I noticed a couple glitches in Amazon orders I made during the “glitch”. Got one item wrong (ordered Magnesium and got Magnesium Oxide) and a couple were delayed and had several updated delivery dates as they got a handle on it.

[Noumenon]

As a retired network engineer, I can attest that all of what you say is true. It doesn't take much to "poison" DNS caches. I've seen too many cases where it's the last place anyone thinks to look. It's a favorite malware target. This doesn't even begin to touch misconfiguration errors at any place along the DNS chain. It can be hard to get someone else to look at their own devices (Cisco, I'm looking at you) in pursuit of finding and correcting the problem.

Late in my career, I became a big fan of the Cradlepoint ecosystem. Their software and devices made it easy to route around most of the common DNS fuckery one would encounter. It did assume that one knew what they were doing. As was too often not the case. IT certs and college classes don't necessarily grant one an intuitive grasp of DNS and IP routing and hardware ecosystems with all of their peculiarities.

I've had to deal with higher-ups who went all ooooh, let's move everything to the cloud. I always had to bring the discussion back to the concept of business continuity. And the question was: how long can you operate the business without cloud connectivity? The same question applied to other critical path systems like servers and firewalls. I was always careful to document the responses to these questions, as I'm sure you can imagine why.