Posted on 10/05/2004 9:24:10 PM PDT by John Robinson
It's always something new with a complex system. We have frontends, backends and databases, not to mention ancillary services like DNS, mail, and internal gadgets. Something is bound to goober up.
A few months ago we were hitting the limits of our database environment. I added hardware and all was good... well, too good. The backends couldn't keep up, so I added hardware (just this weekend!), and all was good... until tonight, when things were once again too good. This time the frontend went on strike, overwhelmed.
A few years ago, when last looking at the scalability of our site, we made a choice, to save bandwidth, we spend extra CPU cycles to compress server responses. We achieve roughly a 60% savings.
Unfortunately, that takes a dear toll on our two little 933 MHz CPUs running the frontend. That toll looks something like this:
<code type="unix geek">
8:15pm up 2 days, 14:48, 11 users, load average: 214.69, 160.02, 84.25 317 processes: 306 sleeping, 10 running, 1 zombie, 0 stopped CPU0 states: 12.1% user, 69.0% system, 0.0% nice, 18.0% idle CPU1 states: 12.0% user, 63.0% system, 0.0% nice, 24.0% idle Mem: 2064712K av, 2050876K used, 13836K free, 0K shrd, 46816K buff Swap: 2040244K av, 22900K used, 2017344K free 223376K cached</code>
Translation:
So... I added more hardware! That explains the second IP address (209.157.64.201, the first is 209.157.64.200.) In a few days your ISPs will have our updated DNS, and will automagically select one of the two frontends when you visit www.freerepublic.com.
The new frontend is a dual 1.4 GHz whopper. It along with it's older partner will have no trouble slinging compressed pages now, and saving roughly 2 grand a month. Oh, and when I said I added more hardware, well, I actually reassigned an older backend to frontend duty. I thought I may have to rearrange machines while I tune the system, so I made it easy ((cough)) to do.
We're running fine now. The peak load I saw was 130 requests per second. I figure we were probably doing 160-180 per second during the debate. No way to know for sure, the fire burned up our logs.
As for the new hardware, I know many people have been asking about it, and how the install went last weekend. I just haven't yet had the time to write what I wanted to write.
In summary, we added three Dell ((cough)) PowerEdge 1750 servers each with dual 2.8 GHz Intel Xeon processors and 1 Gigabyte of RAM. I was really impressed with the Dell machines out of the box, they're mean looking boxes and have more features than the barebones Supermicro kits I used before. Of course, the rails were too short for my rack and there was no table space to lay them. What else can a guy do but rebuild a rack on a Saturday night/Sunday morning? Ah, but that's for another story to tell.
Did you try logging on?
ok...anyone speak nerd, I have no idea what he said and I am somewhat of an amateur techie...
Just don't cut the tag off the mattress. :)
One of our FR boxes melted, he replaced it, the site will be working better now.
Al Gore would be proud!
I had the same problem before I bought a new computer.
I could actually hear the gears grinding. The synchromesh thingie was shot.
The new computer has an automatic transmission. It's easier for me to keep my eyes on the screen.
Hey, If you've got to ask you don't need to know
So what you're saying is, if things go any better there's going to be a small thermonuclear explosion?
Now that's what I was trying to say. Even thought the system is new it broke down and took a little while to figure out why and how to fix it. LOL
I appreciate the alternate server. It is weird, but working fine. Thanks.
Frontend, backend....... this sounds too much like my dad's birds and the bees speech.
LOL, I got it with the picture of the thingy burning.
Dammit,tired of excuses,lets' fix it,make things happen;)
Just a small one.
it is to support upgrades (even in the middle of a cpu fire) that I choose to be a modest monthly contributor.
It's just the right thing to do!
Thanks for the update, and the upgrade... Not having my primary news source for any period of time is hard, but with the upgrades, I know that the downtime was only temporary!
Oh, YES!!! I'm impressed. Definitely. Good job! :-)
Ya done good and Thanks for all your efforts!!!
Looking forward to the next 2 debates and election night and smooth sailing at FR.
Two of those would be describing ME! LOL!!
THANKS for the work you're doing to keep us up & running!!!
Yes, there is some tuning left to do. Tonight was one of the more painful tune-ups. I'm going to try to overwhelm the server tonight, see if I can't knock out any more bugs. I'd have gotten to this bug hunt sooner, but I really wanted to catchup on my Z's. ;-)
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.