Posted on 10/05/2004 9:24:10 PM PDT by John Robinson
It's always something new with a complex system. We have frontends, backends and databases, not to mention ancillary services like DNS, mail, and internal gadgets. Something is bound to goober up.
A few months ago we were hitting the limits of our database environment. I added hardware and all was good... well, too good. The backends couldn't keep up, so I added hardware (just this weekend!), and all was good... until tonight, when things were once again too good. This time the frontend went on strike, overwhelmed.
A few years ago, when last looking at the scalability of our site, we made a choice, to save bandwidth, we spend extra CPU cycles to compress server responses. We achieve roughly a 60% savings.
Unfortunately, that takes a dear toll on our two little 933 MHz CPUs running the frontend. That toll looks something like this:
<code type="unix geek">
8:15pm up 2 days, 14:48, 11 users, load average: 214.69, 160.02, 84.25 317 processes: 306 sleeping, 10 running, 1 zombie, 0 stopped CPU0 states: 12.1% user, 69.0% system, 0.0% nice, 18.0% idle CPU1 states: 12.0% user, 63.0% system, 0.0% nice, 24.0% idle Mem: 2064712K av, 2050876K used, 13836K free, 0K shrd, 46816K buff Swap: 2040244K av, 22900K used, 2017344K free 223376K cached</code>
Translation:
So... I added more hardware! That explains the second IP address (209.157.64.201, the first is 209.157.64.200.) In a few days your ISPs will have our updated DNS, and will automagically select one of the two frontends when you visit www.freerepublic.com.
The new frontend is a dual 1.4 GHz whopper. It along with it's older partner will have no trouble slinging compressed pages now, and saving roughly 2 grand a month. Oh, and when I said I added more hardware, well, I actually reassigned an older backend to frontend duty. I thought I may have to rearrange machines while I tune the system, so I made it easy ((cough)) to do.
We're running fine now. The peak load I saw was 130 requests per second. I figure we were probably doing 160-180 per second during the debate. No way to know for sure, the fire burned up our logs.
As for the new hardware, I know many people have been asking about it, and how the install went last weekend. I just haven't yet had the time to write what I wanted to write.
In summary, we added three Dell ((cough)) PowerEdge 1750 servers each with dual 2.8 GHz Intel Xeon processors and 1 Gigabyte of RAM. I was really impressed with the Dell machines out of the box, they're mean looking boxes and have more features than the barebones Supermicro kits I used before. Of course, the rails were too short for my rack and there was no table space to lay them. What else can a guy do but rebuild a rack on a Saturday night/Sunday morning? Ah, but that's for another story to tell.
Did you try logging on?
ok...anyone speak nerd, I have no idea what he said and I am somewhat of an amateur techie...
Just don't cut the tag off the mattress. :)
One of our FR boxes melted, he replaced it, the site will be working better now.
Al Gore would be proud!
I had the same problem before I bought a new computer.
I could actually hear the gears grinding. The synchromesh thingie was shot.
The new computer has an automatic transmission. It's easier for me to keep my eyes on the screen.
Hey, If you've got to ask you don't need to know
So what you're saying is, if things go any better there's going to be a small thermonuclear explosion?
Now that's what I was trying to say. Even thought the system is new it broke down and took a little while to figure out why and how to fix it. LOL
I appreciate the alternate server. It is weird, but working fine. Thanks.
Frontend, backend....... this sounds too much like my dad's birds and the bees speech.
LOL, I got it with the picture of the thingy burning.
Dammit,tired of excuses,lets' fix it,make things happen;)
Just a small one.
it is to support upgrades (even in the middle of a cpu fire) that I choose to be a modest monthly contributor.
It's just the right thing to do!
Thanks for the update, and the upgrade... Not having my primary news source for any period of time is hard, but with the upgrades, I know that the downtime was only temporary!
Oh, YES!!! I'm impressed. Definitely. Good job! :-)
Ya done good and Thanks for all your efforts!!!
Looking forward to the next 2 debates and election night and smooth sailing at FR.
Two of those would be describing ME! LOL!!
THANKS for the work you're doing to keep us up & running!!!
Yes, there is some tuning left to do. Tonight was one of the more painful tune-ups. I'm going to try to overwhelm the server tonight, see if I can't knock out any more bugs. I'd have gotten to this bug hunt sooner, but I really wanted to catchup on my Z's. ;-)
Pure F'n Magic!!!
Becki
fair nough...
<crawls back into his hole thoroughly Cheney-spanked...hehe.
John, you are a cool dude. Keep it up.
I never understand this computer lingo. Are we supposed to bookmark one or the other? I am so confused!
Ignor that one hardnose poster -
I didn't understand much of it either but I'm glad he(the operator) knows what it is that needs doing when things go wrong - this site is great and it would be a big blow if we lost it, in my opinion -
just my thoughts - take care now -
That looks fun. I have no experience with AMD but have been meaning to give them a try.
Nice Try John, but we all know it was Bush's fault.
Look for Brokejaw to report this tomorrow as another example of Political Jihadery..
On a lighter note. Thanks for all you do in the background to let us, the Pajamahadeen take serious advantage of your skills in getting this site to keep running.
Keep up the great work, and I appreciate you letting us make this a part of our home as well.
Regards,
Joe
Don't touch that dial!
You should take it to one of those shops with the sign:
Computer Spin Balancing
Take your computer in there and have it spin-balanced. That'll take care of the noise.
While we're all speaking techie here, I've been dying to ask you what your database backend is. I assumed it was Oracle. I'm guessing it wouldn't be MS SQL Server for the load. How big is your database?
I'm an experienced & certified Sybase DBA, also working a little with Oracle and Microsoft SQL Server (the latter being the same as Sybase at one time). Hence the curiosity about your database.
We considered the Apple G5 with duals (on FreeBSD) as well, just too much money, and Dell was another $900 or so, wo we went with the Opteron's (just over $3500) with a terabyte of data. If high tech picks up we'll need a number of 'em.
I have no idea what you said but the flaming icon gives me some idea. And I could log into Free Republic tonight when 4 years ago I had trouble getting in on a busy night so I'm happy!
(three Dell ((cough)) PowerEdge 1750 servers each with dual 2.8 GHz Intel Xeon processors and 1 Gigabyte of RAM.)
Damn If I had that much computing power......oh I'd probably waste it playing 3D shooters and downloading .....nevermind.
My Hubby and our son, Darksheare, love AMD. Hubby won't buy anything else. I know nothing about computers but I "know" AMD. I hear it all the time. LOL!!!
That much I understand, and congratulations and thank you.
Congressman Billybob
Latest column, "And the Debate Winner is -- Lemony Snicket"
If you haven't already joined the anti-CFR effort, please click here.
Really, really good job!
I have no clue of what you just said .. but thank you for getting us back up and running
MySQL, it's been good to us and has a choice of storage engines each with different features. We're using the MyISAM storage engine for everything right now, but it's starting to show signs of contention (MyISAM is really good at heavy writing or heavy reading, but not both) so I'll probably start migrating some tables to InnoDB which handles contention better (and does transactions too.)
Oh, the DB is around 30 GB, with another 7 GB of compressed archives which are flat files (actually just HTML.) Those are the older /forum/ URLs you might see from time to time. 2001 and earlier.
Lets have another fund raiser.........last one was too fast !
I understand, a hugh moose bit your sister in the shower.
I was real tempted to install XP and Doom III on one, but I'd still need to acquire the PCI-X vid card and Doom III. Plus I really didn't have the time, if I did, I probably wouldn't have given the machine up. :-)
so, the external clastoid mastoid hyberchronifiar, dismachifnegated the hypostatic ekenosinator, thereby vitiatimating the vortocuticlastical shmagtrofinator...right?
:o)
This is kinda scary I actually understood what you said.
Thanks for being so speedy on the repair.
Thanks for the info. A 30gb db. The biggest single db of the many sybase servers I work with is a mission-critical one about 45 gb, but we have bigger ones in Oracle, which I don't work with. Interesting you have MySQL for a large db and for such a heavily hit application (FR). I'm surprised but impressed. I have to do some research on MySQL dbs.
Wow! That is some load. I have only seen that on an overloaded multinode Nagios monitoring system with 7,000+ checks. But this shows the power of Unix/Unix-like OSes. They might get overloaded but they don't go down. Throw in a little more hardware and a foundry switch to balance it out and you are golden.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.