Posted on 10/05/2004 9:24:10 PM PDT by John Robinson
It's always something new with a complex system. We have frontends, backends and databases, not to mention ancillary services like DNS, mail, and internal gadgets. Something is bound to goober up.
A few months ago we were hitting the limits of our database environment. I added hardware and all was good... well, too good. The backends couldn't keep up, so I added hardware (just this weekend!), and all was good... until tonight, when things were once again too good. This time the frontend went on strike, overwhelmed.
A few years ago, when last looking at the scalability of our site, we made a choice, to save bandwidth, we spend extra CPU cycles to compress server responses. We achieve roughly a 60% savings.
Unfortunately, that takes a dear toll on our two little 933 MHz CPUs running the frontend. That toll looks something like this:
<code type="unix geek">
8:15pm up 2 days, 14:48, 11 users, load average: 214.69, 160.02, 84.25 317 processes: 306 sleeping, 10 running, 1 zombie, 0 stopped CPU0 states: 12.1% user, 69.0% system, 0.0% nice, 18.0% idle CPU1 states: 12.0% user, 63.0% system, 0.0% nice, 24.0% idle Mem: 2064712K av, 2050876K used, 13836K free, 0K shrd, 46816K buff Swap: 2040244K av, 22900K used, 2017344K free 223376K cached</code>
Translation:
So... I added more hardware! That explains the second IP address (209.157.64.201, the first is 209.157.64.200.) In a few days your ISPs will have our updated DNS, and will automagically select one of the two frontends when you visit www.freerepublic.com.
The new frontend is a dual 1.4 GHz whopper. It along with it's older partner will have no trouble slinging compressed pages now, and saving roughly 2 grand a month. Oh, and when I said I added more hardware, well, I actually reassigned an older backend to frontend duty. I thought I may have to rearrange machines while I tune the system, so I made it easy ((cough)) to do.
We're running fine now. The peak load I saw was 130 requests per second. I figure we were probably doing 160-180 per second during the debate. No way to know for sure, the fire burned up our logs.
As for the new hardware, I know many people have been asking about it, and how the install went last weekend. I just haven't yet had the time to write what I wanted to write.
In summary, we added three Dell ((cough)) PowerEdge 1750 servers each with dual 2.8 GHz Intel Xeon processors and 1 Gigabyte of RAM. I was really impressed with the Dell machines out of the box, they're mean looking boxes and have more features than the barebones Supermicro kits I used before. Of course, the rails were too short for my rack and there was no table space to lay them. What else can a guy do but rebuild a rack on a Saturday night/Sunday morning? Ah, but that's for another story to tell.
Ooops, sorry! Meant John! (I'm used to thanking Jim -- but THANK YOU JOHN!) Sheesh! I need more coffee.
Yeah, I know the feeling. My backend is doing just fine, but my frontend needs enhancement.
That's no longer an issue or it isn't an issue with the right setup. We researched a number of issues for a couple of weeks, especially the cooling issues but our concerns were laid to rest. The SuperKool technology keeps the 1u boxes cool - the 2,3 and 4u units stay about the same temperature.
Four years ago I had this setup at home. After too many warm days we moved to a real data center with redundant everything. Heat has always been an issue for us but we're no longer concerned with the AMDs.
"317 processes: 306 sleeping, 10 running, 1 zombie, 0 stopped"
We found where that missing Zombie went.
John's got him and put him to work!
"The breakers blew twice here at home when I was compiling Gentoo on all three boxes. My 650 watt UPS powered them along for literally 5 seconds before it gave up in disgust. LOL"
Oh man, that's sadistic!
Compiling Gentoo.. ad then having the UPS power the things for 5 minutes?
Interesting. We've used Intel for workstations and had problems and AMD for servers and never (since 99) had problems. Well, the servers are getting old...
Swap: 2040244K av, 22900K used, 2017344K free 223376K cached
Gee, you're hardly touching your swap space.. what's the problem?
CPU0 states: 12.1% user, 69.0% system, 0.0% nice, 18.0% idle CPU1 states: 12.0% user, 63.0% system, 0.0% nice, 24.0% idle
Oh, yeah.
I have three amateur radio repeaters in a 10-foot-tall cabinet at a local broadcaster's transmitter site. No air conditioning in the transmitter room. And we wondered why things sounded really distorted on a hot summer day... especially after I installed EchoLink, which keeps the repeaters quite busy. I installed a temperature sensor and found the in-cabinet temp to be around 120 to 130° F!
I had to install a big hassock fan outside the cabinet to blow cool air in, as the radio fans were just circulating the hot air around inside the cabinet. Now the cabinet stays within 5 or 6 degrees of the room air temperature, which still gets up around 105° or so on a hot day. The fact that we're right next to the NOAA Weather Radio transmitter cabinet doesn't help.
Gotta keep those things cool, man!
The table was nice, but I needed the flaming tower icon to understand it. That new frontend was the bees knees last night. When the traffic is automatically divided between the two, I suspect everyone here will be smiling broadly.
Seconds even. Probably less than 5, maybe half a second. By the time I realized only a breaker blew (the rest of the house was powered) the UPS shut down. I didn't drain the battery, I think I was drawing too much power, must have tripped the UPS' breaker too. LOL!
I got real good at compiling and installing Gentoo (stage1 no doubt.) Ended up building six sets of packages (PentiumII, PentiumIII, PentiumIV for server installs, and the same for workstation installs) and bunches of kernels for the various classes of systems I use. Made tarballs of everything, and am now working on tarballs of configured systems for the various machines I want to upgrade (from RedHat 7.2--cringe.) When they're ready, all I'll need to do to upgrade a machine is to boot the LiveCD, fdisk, mke2fs, untar, grub-install, and reboot ready to go. Very happy.
Soooo, what are you going to do in your spare time?
OUCH.
Sounds like my "three minutes until oblivion" Office UPS.
In reality I have to crash dump things to shut down in under a minute.
The power company here likes to play little voltage games, so we call them "Spike and Bite", because the voltage in the lines spikes and then drops off the scale.
Friend of mine swears by RedHat 7.2...
*scratches head*
LOL!
When you get to doing the upgrades on the project machines, good luck.
And have fun.
*chuckle*
right click 'save target as' Zorak on strike, save image as Zoraksign.
using intel mobos for the workstations? intel makes the worst boards ever in my opinions...(and that of many many many others)
go for an asus for workstation or supermicro for server. same price, much better support and compatibility.
$124.63.
Thanks, John.
Makes perfect sense to me, yep. ;-)
Thanks for the ping, tame. It's a humbling experience to realize that I basically know nothing about how this whole internet thing works. LOL
BTW, when I spell-checked this post, "clastoid", "hypostatic", and "ekenosinator" came up as the only words in your sentence spelled wrong. "Shmagtrofinator" is a word??? LOL
Uh oh. Too late for that! I may disappear into never neverland when this all happens, whatever this is that is going to happen! :-)
:o)
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.
