Years ago, I was working in a datacenter that had about 13, HP-3000s with about 120 or so big washtub disk drives strung out the back on the floor. We also had a water-cooled IBM 3090, and a couple of miscellaneous Dec 11/780s. With all these systems up and running, this was a loud room to work in.
One day we had some fine fellows hanging wallpaper in the corridor that connected the computer room from the secure area you had to go through to enter.
There was a Big Red Button on the wall that had the words "EMERGENCY POWER CUT OFF" printed over in it large red letters. This button also had a cover over it that had to be pulled up in order to press the button, so noone could bump it by accident.
You remember those fine fellows hanging wallpaper? Well, they had to take that cover off the BRB so they could hang their wallpaper.
It was about 4:30 or so and we were right in the middle of shift change. A bunch of us were standing around talking and passing on info about what had happened the previous shift and what was coming up. Suddenly, we heard a huge BANG and it went dark as we heard the slow wind-down of all the fans, drives, and computers.
It became really quiet in that room. A kind of quiet you seldom hear, as our ears were so accustomed to the drone of the fans and drives, the lack thereof was even more profound than it might otherwise had been. We all kind of looked at each other and then looked out to the corridor, and saw where one of the fine fellows hanging wallpaper had accidently brushed up against that Big Red Button.
The aftermath was kind of interesting. We went through, hit the individual power switches on all the disk drives and other peripherals, to turn them off, then brought power back on.
Once we were sure power was stable, we started turning on the drives to the HP-3000 boxes. Out of the 12 systems, (that had thousands of users), 9 of them just kinda sat there and waited until each of its drives were fully up and connected, then they started executing the next instruction in their stack. I didn't know it at the time, but apparently they contained their own battery backup in the chassis, and just waited until they could continue right on with their work! To the thousands of users connected to these systems, their terminals just froze for a while, then continued right where they had left off. Total downtime for most of these systems was 15-30 minutes. (This represented thousands of hours of cumulative user downtime.)
The 3090 didn't fare so well. Apparently, they don't particularly like it when someone just yanks power from underneath them. Took about 13 or so hours to get it fully operational again.
I'll never forget what the Sound of Silence is really like.
I was installing a medium-sized cluster for a client. The VP of my company had decided to tag along to see what we did in the field. I didn't mind, as I was pretty good at my job, and I had been to this client several times previously. We all got along.
So we arrive on site and start assembling this thing (I typically started from boxes, and ended up with a usable machine), and it took a few days. In the meantime, I had noticed that the power being supplied to me was insufficient, and pointed it out to the client. He got facilities there, and they changed out the power connectors for me while I continued working on the system.
I plugged everything in, and turned on the system.
10 minutes later the power to four buildings (including the one I was in) went down hard. Turns out that the power panel we were connected to was already slightly overloaded before we got there. Adding this power drain was just too much.
Wasted a day and a half to get things right again. :)
...And the VP finally got to understand why I claimed that something always goes wrong at these installs--and never the same thing twice.