New Numbers for the Curious (or FR Traffic Stats)

Free Republic Website Usage ^

[John Robinson]

For those that love numbers, I have three links to Free Republic usage statistics:

• Free Republic Website Usage NEW!
  compiled from server logs; ready by 2am PT daily

• Free Republic Daily Stats
  compiled from posting logs; ready by 2am PT daily

• Traffic Rank [Alexa]
  see their website

Caveat:

1. I recently upgraded the program (AWStats) used to generate the Website Usage pages. The old pages, though built with the new program, utilized the old format database which did not provide some fields. Those fields will be 0 or not present on the older reports.

2. For some reason the reports switched to GMT some months ago, I switched back to PT this month. Each switch fouled the "Hours" graph for the month of the switch.

3. We lost our senses Mar 12-14, or rather, the stat program forgot how to tell the difference between a Hit (any request to the web server) and a Page (a request for a HTML page, or similar page of content); all hits were erroneously counted as pages, the actual number of pages for that time period is actually quite a bit less.

4. And, on Wednesday Mar 12th, I enabled HTTP compression, practically halving our bandwidth usage! Just in time for the overwhelming load that day!

5. The Daily Stats page is clearly showing signs of age. It no longer collects the "Front-Side" and "Back-Side" statistics. However, you will find "Front-Side" had become a duplicate effort, see the Website Usage pages to continue on.

6. Numbers on the Daily Stats page and the new Website Usage pages are generally an even comparison. However, consider the Daily Stats pages are generated on daily GMT time boundaries, and the Website Usage pages are generated at PT (or GMT--oops) boundaries.

And now the ramble.

One thing I don't yet have for readers is a report of the actual Mbps that flows through our Internet connection. Leading up to the war we've routinely saturated our link, that's 10 Mbps, or 10 million 1's and 0's per second. Those numbers are very interesting as well!

As we reach our bandwidth limit, the network link becomes saturated and we begin to slow down. Last Wed (Mar 12) was the breaking point, the server was falling over as network congestion backed up through the kernel and into my application-- what a mess that will make!

Foreseeing impending trouble days before, I had previously installed a little gadget called mod_gzip. The mod_gzip is an Apache webserver extension that compresses HTML pages on the fly. After an abbreviated test run of just a few days I enabled mod_gzip to bail me out of this crisis.

My savior! The compression is highly effective, reducing pages by an average of 79% the original page size. However, the true savings are closer to 50%. Whether the module is used at all is dependent upon the browser used to view the page. Due to brain damage, some browsers do not work properly or at all when viewing a compressed page, and therefore must not be sent compressed pages. And images are not compressed at all-- mostly because images are already highly compressed, but partly because of even more brain damage in even more browsers.

50% isn't bad, not bad at all. We've now effectively doubled our bandwidth capacity. Have we tamed the traffic? Mostly. Bandwidth usage is now routinely in the 5-6 Mbps range, which is effectively 10-12 Mbps. Thursday morning from 7am-1pm PT we were in the 7-8 Mbps range (14-16 Mbps!) and often times pushing the ceiling, pushing our full link capacity of 10 Mbits hard for extended periods of time. That's 20 Mbits of bandwidth, 12 T-1 lines, or nearly one half T-3. Simply amazing!

[RottiBiz]

Same thing happened to me. Click the red "source" at the top of the page and you will get this:

http://www.freerepublic.com/usage/awstats/

Worked for me.

[Seeking the truth]

I think John is saying "We're doing something good".

[John Robinson]

Oops! I posted the wrong URL. I corrected it in the article body, here it is.

I have two versions of that report. One has administrative information (IP addresses, etc.) At least I now know the access controls work.

[BureaucratusMaximus]

How about you come up with some new stats like:

Number of ZOT posts

Number of ZOT posts Zotted

Estimated time from the initial ZOT posting to when the post is ZOTed

Also...I havent seen the Viking Kitty pic in a while...and I'm quite depressed LOL.

[HairOfTheDog]

Due to brain damage, some browsers do not work properly or at all when viewing a compressed page, and therefore must not be sent compressed pages.

I hope I am not working on one of the brain damaged browsers!

[txzman]

Amazing Stats. And a great reminder that people need to help support FR with MONTHLY payments.

Liberty ain't cheap!

[John Robinson]

Dual CPU, dual server. We have two web servers. They're reached through a reverse proxy. The proxy does the compression. It really is a sight! The proxy currently reports a .122% CPU load at 57 requests per second average. And since most pages are generated and dumped into the kernel buffer to be spoonfed to the network (and the file descriptor handed off to a third party for lingering close) the web daemon is quickly freed from the request. During heavy load the daemon often only has a few requests being served at any moment in time.

(But of course, as I sit here bragging about how well it is running, it all falls down. Oh!)

[nicmarlo]

ty......I'm glad to know it wasn't just ME that was "unauthorized," lol! : )

[nicmarlo]

That's the most interesting link of all (the one you fixed), IMO.....thanks for your post and fixing the link, John....

[TontoKowalski]

New signups on Thursday: 350...the fourth highest total behind 3 days in the timeframe just after the 2000 elections.

Welcome, newbies! Glad to have you!

And now, we consecrate the bonds of obedience.

[FourtySeven]

Translation, for me: Rowr-rowr-rowr-rowr rowr, rowrrowr, rowr, rorrworwor. (like from the Charlie Brown cartoons, when a teacher or other adult would talk to one of the kids)....hehe

I'm just glad you understand it....keep up the good work Misters Robinson!

[Tamzee]

Wonderful to see :-)

[Salvation]

BTTT!

And congratulations on the new server working so well.

[Alas Babylon!]

WAY KOOL! You got a server, or system of servers, that's for sure!

[PA Engineer]

Thanks and bump.

[John Robinson]

grania--
It was removed because of a severe performance penalty due to each hit causing a database update. I have, in theory, a better way of doing it now, I'd like to try it again later.

FreedomPoster--
I would guess, from the telco's perspective, voice and digital circuits are similar if not one-in-the-same, and the pricing would probably be similar as well. I don't really know, I have neither. We're colocated with Verio in San Jose. We don't need no stinkin' CSU/DSU or other telco headaches.

tictoc--
HTTP 1.1 is probably the default on capable browsers. I haven't ever needed to switch it on, but somebody may. Thanks for pointing that out.

---

It's been a lot of fun getting this system tuned to handle the war time traffic. I've learned much about the guts of the technology we use here. I cried when I realized there was nothing left to tune. :-D We need more traffic! More tuning! MORE TUNING!

[tictoc]

Apparently mod_gzip has been around for a while, but many websites have not yet implemented it.

This page -- Making your website super fast-loading -- was not too far above my head, and it contains links to more detailed information.

[new cruelty]

Thanks for the stats. I enjoy a good read!

[John Robinson]

HTTP compression is a thorny issue. So far I've had only one report of a reader unable to visit. OTOH, without it, thousands would be unable to visit. It's time to upgrade those browsers.

[John Robinson]

Now:

Sustained bandwidth of 8 Mbits, peaking at 10 Mbits. 70 requests per second, often seeing 80 and 90 requests per second. No sweat. Give me more!