I don’t believe that John is making backups. Even if the data center had been hit by a nuke, it would not have taken more than a day to do a bare metal restore from backup onto new hardware. Also, I suspect that none of the machines in question are running RAID. I have individual drives fail from time to time in servers and it’s no big deal—all are hot-swappable and it’s just a matter of plugging in a new drive and telling the array to rebuild—no downtime required.
Also, my comments are predicated upon the assumption that Jim R. does not want to move to cloud-based servers, and prefers to own and colocate his own equipment. I can understand wanting to do this considering this site’s content; however, it’s obvious that both the choice of hardware and disaster recovery procedures are totally inadequate. Hell, the admin’s laptop isn’t working, so he can’t do the job? WTH
You have to assume that drives WILL fail from time to time... and SSD's aren't immune to failure either.
Back when I was doing this, we always kept spare drives in the cage with the servers. When needed, we could call the COLO staff and tell them to put a new drive into bay XXX and they would... then we remotely configure it and bring it online. Simple.
Try deleting the first '3' in the current thread's URL.
If John is not making backups, that should get you a 404. Or take you back to 2001 if he is.