What happened? Was it a directed attack, or a server fart?
It’s a fundraising tactic. Lol...
I just got off the phone with John. He thinks he finally found the problems and fixed them. It was a compound error in the system configuration which only occurred under a heavy load and was very tough to spot. I asked him to stop in and explain it himself. Hopefully he will. Also, hopefully, the problem is fixed or at least he has an inkling of where to look next.
Ended up being a complicated little mess. Problem was the database server would stop answering network connections, causing all the web clients to get stuck. It took awhile to figure out why. It turns out that the DB was configured to do DNS lookups on client connect. Because the DNS service is running on a very busy machine, random DNS queries were getting dropped. Apparently there’s some kind of bug in the DB that wouldn’t timeout a connection if it didn’t get a DNS response, so eventually the whole system gummed up and stopped responding. 0 activity, no system load, and just zilch.
Only showed up when the site was very busy. I disabled DNS lookups on the DB and it seems to be okay now. We’ll see how it goes.