Free Republic

Just saw that you switched from Apache to Nginx. +1. Good luck tomorrow, FP.

Another few suggestions:

1. Install “htop” to see where all the memory, CPU, and load averages is going. Once you’ve found a process you’d like to trace, simply his “s”.

2. Check to see where the memory in apache processes are coming from. Specifically, use “top” and “lsof” to see the top apache processes. Once you’ve found one that seems to be about average, install “lsof” and type “lsof -p {pid}”, where {pid} is the process ID of the apache process you want to diagnose.

3. I’ve had where my access log was literally being loaded into my apache processes each and every time. With high-trafficked websites, that’s wasted memory per process. You can quickly see if that’s a problem using “lsof”.

4. Switch FreeRepublic’s DNS to use CloudFlare. This way even if the site goes down, everyone has access to the site’s content and information. It’ll also help filter out bad and aggressive DDoS attempts. Less bad requests = faster FR.

Since you’re ears wide open, I’ll contribute some thoughts:

If the problem is with people having to wait in a queue, and since your page headers reveal you’re using Apache, then the problem is most likely on the Apache layer. It’s difficult solve problems without a few answers, but I might suggest finding out or consider the following:

- How many servers is Free Republic being served across? If it’s only one, how many GB of RAM does it have and how many cores?

- Users are forced to wait in queue most likely because the Apache directive of MaxClients or ServerLimit is reached.

- The best way to increase accessibility is to find out how much memory the maximum apache process is using (assuming you’re using Apache 1). Then review each module that’s loaded into apache. Apache’s physical memory dedicated to processes can be reduced if you remove unneeded modules, and thus allowing you to increase the max number of clients that receive the request.

- If the site is served across multiple servers, then is it on a load balancer? While scaling vertically can have a lot of added benefit, scaling horizontally will help alleviate the immediate problem while focusing on the core.

- Is the database using MySQL? If so, is it using InnoDB, MyISAM, or NDBC?

- Is the database setup in a cluster environment with redundancy, or is it set to read across the site mostly from read-only slave databases and only the master has writes?

- How many Max Children process are set for each apache process?

- When querying the data on individual post or main pages, are you limiting the data to only the columns needed? That can save significant time in searching and retrieving data (eg: excerpt instead of full_content and excerpt). Avoid SELECT * like the plague (you can even see that in the main URL of the Latest Posts, ouch).

- Is the database using primary keys to help index the content? Seems basic, but Free Republic is ancient.

- Patch up sever memory leaks and holes. I can name one off hand. http://www.freerepublic.com/perl/resend is a page that if I submit to it, it takes 15 seconds to even return a response while the rest of the site is handled quicky (good internet). If someone were to run a script which targeted that page alone, they’d probably be able to shut down the server with ease. (eg: simply type in the word “admin” in the username field and hit enter). See for yourself. With full confidence, a botnet of over 300 computers could wreck Free Republic in less than 5 minutes. Not including hidden security exploits that lurk in your site you’re probably not aware of, like your RSS engine that should be checked on. (eg: http://goo.gl/7NVIq)

- TURN GZIP ENCODING ON AND MOD_DEFLATE ON. If you’ve got the CPU cores, turn it on. At compression level 0 (what it’s at), the Latest Posts page is 69,855 bytes, or 69kb. With mod_deflate on at compression level 5 (goes up to 9), it turns to 21,595 bytes, or 21kb. A 69.1% percent bandwidth savings that would be applied across the site. Faster deliverying pages, means faster loading on mobile devices and faster transfering to slow connections. Faster delivery means faster closing connections to apache, which means you can concurrently handle more connections.

- Caching. Now THAT’S a scary word for Free Republic. Or is it? Here’s an idea. If you go to any post page, or even the home page, you’ll notice there’s no dynamic content. All time stamps are fixed at the time they’re posted, the unique ID increment of a comment is fixed once it’s pushed through. How about for Anonymous or Guest users who aren’t registered, delivery all pages on a 3 minute cache timer. That allows you to decrease your server load DRAMATICALLY, in fact, if you made ANY change that had the most impact, that’d be it. That’s a low enough number to still be fresh, and not high enough to be stale.

- Optimize MySQL to increase table cache, buffer, query size, etc. There are automated tools that can recommend settings specific to your server configuration that’s a one-click download.

- One of the biggest slow-down of Apache is server media files. But wait, I thought Free Republic doesn’t allow image uploading! Right, but Free Republic does server a few very important files. “/l/common.css” and “/favicon.ico”.

- Did you know the number one most requested file of any site is usually the Favicon.co file? It seems so innocent, but if optimized, can save hundreds of gigabytes in server bandwidth a month for high-trafficked websites such as FR. Currently the favicon is 32x32 at 2.2kb. The standard size is 16x16, which makes the favicon come out to 1.4kb. Even if we’re modest and suggest that Free Republic gets 1,000,000 unique views a month, that’s 781gb of data saved. With that file ALONE.

- While we’re on it, we can actually tell Apache to NOT server media files, but instead tell something that’s solid and powerful like memcached, a memory-mapping engine to server them instead. Once again, lowering the memory needed and increasing the number of concurrent connections.

- Anywhere on the site where FR has to query all users to return a result, it fails and crashes, that’s an immediate point of target for DDoSing. Does FR implement a hard firewall or an automated Firewall system that null route / blackholes attackers?

- Without changing a single thing in the site, just looking at what “slow” operations perl is doing could save milliseconds per page. Milliseconds over a few hundred people means people are getting pages faster in seconds. http://www.ibm.com/developerworks/library/l-optperl/index.html

- Changing how your browser downloads content is one solution, but the fault is on the server, not the user. Throwing money at the problem

- Install Newrelic for 1 hour to collect debugging information on the server. That’ll identify every problem you have and need fixed.

- I also suggest moving away from Apache, and use Lighttpd, as it uses less memory, less CPU time, and is significantly faster. Even Nginx is better.

- The good news is that your email hosting is on a different IP address than where FR is hosted, which is great.

Jim and John, you’ve got options. I’m sure this has all been mentioned before. There has never been a more critical time to pursue one or many of these options. You take pride in closing the world out to FR’s workings to protect it. That’s okay, but that means the future rests solely on your shoulders. You have an enormous amount of people who rely on you. It would be my suggestion to listen to their suggestions, something many suggest our current leaders aren’t doing today.

Best regards,
Tim Selaty Jr., Co-Founder of Tea Party Tribune
timselaty@gmail.com