Well, accessing FR from a different IR address suggests to me that there has been some kind of DNS server problem (almost always explained by an attack).
I look forward to the post-mortem on this incident, because I am educated by each case study.
Infrared address????
I think it was a load problem - allowing 250 posts per page was way too many. And when people weren't getting anything, they kept refreshing which just made things worse.
That's a second frontend server I brought online. We'll have two frontends after this, 209.157.64.200 and 209.157.64.201. New bottleneck was in the frontend system.
So that about covers it. Added a fifth server to help the database, which was the first bottleneck. With the database flying, the backend servers were the bottleneck, added three more to fix that. Now the frontends are the bottleneck, so I reassigned a machine to help out. I also experiemented with turning off the CPU expensive compression, which proved that bandwidth was the bottleneck. Ack! Very frustrating.