Posted on 10/05/2004 9:24:10 PM PDT by John Robinson
It's always something new with a complex system. We have frontends, backends and databases, not to mention ancillary services like DNS, mail, and internal gadgets. Something is bound to goober up.
A few months ago we were hitting the limits of our database environment. I added hardware and all was good... well, too good. The backends couldn't keep up, so I added hardware (just this weekend!), and all was good... until tonight, when things were once again too good. This time the frontend went on strike, overwhelmed.
A few years ago, when last looking at the scalability of our site, we made a choice, to save bandwidth, we spend extra CPU cycles to compress server responses. We achieve roughly a 60% savings.
Unfortunately, that takes a dear toll on our two little 933 MHz CPUs running the frontend. That toll looks something like this:
<code type="unix geek">
8:15pm up 2 days, 14:48, 11 users, load average: 214.69, 160.02, 84.25 317 processes: 306 sleeping, 10 running, 1 zombie, 0 stopped CPU0 states: 12.1% user, 69.0% system, 0.0% nice, 18.0% idle CPU1 states: 12.0% user, 63.0% system, 0.0% nice, 24.0% idle Mem: 2064712K av, 2050876K used, 13836K free, 0K shrd, 46816K buff Swap: 2040244K av, 22900K used, 2017344K free 223376K cached</code>
Translation:
So... I added more hardware! That explains the second IP address (209.157.64.201, the first is 209.157.64.200.) In a few days your ISPs will have our updated DNS, and will automagically select one of the two frontends when you visit www.freerepublic.com.
The new frontend is a dual 1.4 GHz whopper. It along with it's older partner will have no trouble slinging compressed pages now, and saving roughly 2 grand a month. Oh, and when I said I added more hardware, well, I actually reassigned an older backend to frontend duty. I thought I may have to rearrange machines while I tune the system, so I made it easy ((cough)) to do.
We're running fine now. The peak load I saw was 130 requests per second. I figure we were probably doing 160-180 per second during the debate. No way to know for sure, the fire burned up our logs.
As for the new hardware, I know many people have been asking about it, and how the install went last weekend. I just haven't yet had the time to write what I wanted to write.
In summary, we added three Dell ((cough)) PowerEdge 1750 servers each with dual 2.8 GHz Intel Xeon processors and 1 Gigabyte of RAM. I was really impressed with the Dell machines out of the box, they're mean looking boxes and have more features than the barebones Supermicro kits I used before. Of course, the rails were too short for my rack and there was no table space to lay them. What else can a guy do but rebuild a rack on a Saturday night/Sunday morning? Ah, but that's for another story to tell.
Pure F'n Magic!!!
Becki
fair nough...
<crawls back into his hole thoroughly Cheney-spanked...hehe.
John, you are a cool dude. Keep it up.
I never understand this computer lingo. Are we supposed to bookmark one or the other? I am so confused!
Ignor that one hardnose poster -
I didn't understand much of it either but I'm glad he(the operator) knows what it is that needs doing when things go wrong - this site is great and it would be a big blow if we lost it, in my opinion -
just my thoughts - take care now -
That looks fun. I have no experience with AMD but have been meaning to give them a try.
Nice Try John, but we all know it was Bush's fault.
Look for Brokejaw to report this tomorrow as another example of Political Jihadery..
On a lighter note. Thanks for all you do in the background to let us, the Pajamahadeen take serious advantage of your skills in getting this site to keep running.
Keep up the great work, and I appreciate you letting us make this a part of our home as well.
Regards,
Joe
Don't touch that dial!
You should take it to one of those shops with the sign:
Computer Spin Balancing
Take your computer in there and have it spin-balanced. That'll take care of the noise.
While we're all speaking techie here, I've been dying to ask you what your database backend is. I assumed it was Oracle. I'm guessing it wouldn't be MS SQL Server for the load. How big is your database?
I'm an experienced & certified Sybase DBA, also working a little with Oracle and Microsoft SQL Server (the latter being the same as Sybase at one time). Hence the curiosity about your database.
We considered the Apple G5 with duals (on FreeBSD) as well, just too much money, and Dell was another $900 or so, wo we went with the Opteron's (just over $3500) with a terabyte of data. If high tech picks up we'll need a number of 'em.
I have no idea what you said but the flaming icon gives me some idea. And I could log into Free Republic tonight when 4 years ago I had trouble getting in on a busy night so I'm happy!
(three Dell ((cough)) PowerEdge 1750 servers each with dual 2.8 GHz Intel Xeon processors and 1 Gigabyte of RAM.)
Damn If I had that much computing power......oh I'd probably waste it playing 3D shooters and downloading .....nevermind.
My Hubby and our son, Darksheare, love AMD. Hubby won't buy anything else. I know nothing about computers but I "know" AMD. I hear it all the time. LOL!!!
That much I understand, and congratulations and thank you.
Congressman Billybob
Latest column, "And the Debate Winner is -- Lemony Snicket"
If you haven't already joined the anti-CFR effort, please click here.
Really, really good job!
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.