Posted on 10/31/2012 6:35:41 PM PDT by Jim Robinson
Very sorry about the problems with our servers causing FR to get bogged down under peak traffic loads. Praying that John finds a solution or workaround soon.
In the meantime, here are some suggestion you can use to possibly make your FR pages load faster:
Select "Disabled" in the "Region" pulldown for each sidebar block you wish to temporarily disable then click "Modify":
http://www.freerepublic.com/perl/manage-blocks
http://www.freerepublic.com/perl/settings
And it’s simply a click to toggle "Brevity" from "Text" to "Headers" or back the other way when needed:
http://www.freerepublic.com/tag/news-forum/index
If enough of us did this we would reduce the amount of data downloaded on each click, reducing the load on the database and the server and possibly increasing overall systems performance.
To reset your preferences next week after Obama is evicted and the tea party tsunami sweeps the socialists out of office from sea to shining sea, click on "Account" at the top of any of the main index pages and then "Manage Blocks" to configure your sidebars and "My Preferences" to reset your number of threads and replies per thread preferences.
http://www.freerepublic.com/focus/my/
Also, we have FR contingency standby pages on yahoo and facebook. You may wish to visit and bookmark these pages:
http://www.facebook.com/groups/26012226159/
http://groups.yahoo.com/group/freerepublic2/
Our system didn't actually crash during the debates, but when a hundred thousand or more users tried to access it all at once during a short two hour window, it bogged down to a crawl. Hopefully it won't do that on election day as the window will not be so short. The news and results will be coming in all day and all night. But if it does crash, we can leave messages on the contingency sites linked above.
Thank you all very much and good luck.
Prayers up for our nation.
Because you are a 2nd amendment sister and in TX. Nothing will mess with you!
A little late to the party, but thanks Jim.
I think if we all dropped our sidebars for the duration that would make it less of a demand as well.
20 latest or 50 replies and nothing else.
Thanks Jim.
I use this link...
http://www.freerepublic.com/tag/*/index?brevity=brief;tab=comments
Is there a way to filter out the bloggers and personal posts?
POP
bookmark
Thanks Jim ... I too reduced/changed all my account preferences to reduce the load on our FR system. Wisconsin is Going RED! — ryn
How about buying a server that can actually perform to your needs? You seem to be taking in enough do-re-mi to make the purchase. Or if it’s a software problem, get an IT pro to square it out? This site is just frustrating to try and carry out a conversation on. If it wasn’t for so many like-minded people I doubt I’d put up with it.
Jim, you probably get a lot of advice on this.
I was wondering if it’s a DNS problem, where the signal is bouncing all over the web trying to get to your servers. It may have to do with servers having been altered or replaced over the years. I believe it’s relatively easy to fix.
Regardless, I was here at the beginning, and suspect I’ll be here at the end.
Whatever happens Tuesday, FR continues the good fight.
Thanks for all that you and John do.
- FRegards
I turned off my state notifications - seems like that could be added to the list.
How about forcing as many of these “suggested” changes from today through a day or two after the election?
Why leave it up to the users at this point?
This would be the same as a commanding officer enforcing rationing of limited resources during a time of crisis, rather than just suggesting everyone use “less”.
Since you’re ears wide open, I’ll contribute some thoughts:
If the problem is with people having to wait in a queue, and since your page headers reveal you’re using Apache, then the problem is most likely on the Apache layer. It’s difficult solve problems without a few answers, but I might suggest finding out or consider the following:
- How many servers is Free Republic being served across? If it’s only one, how many GB of RAM does it have and how many cores?
- Users are forced to wait in queue most likely because the Apache directive of MaxClients or ServerLimit is reached.
- The best way to increase accessibility is to find out how much memory the maximum apache process is using (assuming you’re using Apache 1). Then review each module that’s loaded into apache. Apache’s physical memory dedicated to processes can be reduced if you remove unneeded modules, and thus allowing you to increase the max number of clients that receive the request.
- If the site is served across multiple servers, then is it on a load balancer? While scaling vertically can have a lot of added benefit, scaling horizontally will help alleviate the immediate problem while focusing on the core.
- Is the database using MySQL? If so, is it using InnoDB, MyISAM, or NDBC?
- Is the database setup in a cluster environment with redundancy, or is it set to read across the site mostly from read-only slave databases and only the master has writes?
- How many Max Children process are set for each apache process?
- When querying the data on individual post or main pages, are you limiting the data to only the columns needed? That can save significant time in searching and retrieving data (eg: excerpt instead of full_content and excerpt). Avoid SELECT * like the plague (you can even see that in the main URL of the Latest Posts, ouch).
- Is the database using primary keys to help index the content? Seems basic, but Free Republic is ancient.
- Patch up sever memory leaks and holes. I can name one off hand. http://www.freerepublic.com/perl/resend is a page that if I submit to it, it takes 15 seconds to even return a response while the rest of the site is handled quicky (good internet). If someone were to run a script which targeted that page alone, they’d probably be able to shut down the server with ease. (eg: simply type in the word “admin” in the username field and hit enter). See for yourself. With full confidence, a botnet of over 300 computers could wreck Free Republic in less than 5 minutes. Not including hidden security exploits that lurk in your site you’re probably not aware of, like your RSS engine that should be checked on. (eg: http://goo.gl/7NVIq)
- TURN GZIP ENCODING ON AND MOD_DEFLATE ON. If you’ve got the CPU cores, turn it on. At compression level 0 (what it’s at), the Latest Posts page is 69,855 bytes, or 69kb. With mod_deflate on at compression level 5 (goes up to 9), it turns to 21,595 bytes, or 21kb. A 69.1% percent bandwidth savings that would be applied across the site. Faster deliverying pages, means faster loading on mobile devices and faster transfering to slow connections. Faster delivery means faster closing connections to apache, which means you can concurrently handle more connections.
- Caching. Now THAT’S a scary word for Free Republic. Or is it? Here’s an idea. If you go to any post page, or even the home page, you’ll notice there’s no dynamic content. All time stamps are fixed at the time they’re posted, the unique ID increment of a comment is fixed once it’s pushed through. How about for Anonymous or Guest users who aren’t registered, delivery all pages on a 3 minute cache timer. That allows you to decrease your server load DRAMATICALLY, in fact, if you made ANY change that had the most impact, that’d be it. That’s a low enough number to still be fresh, and not high enough to be stale.
- Optimize MySQL to increase table cache, buffer, query size, etc. There are automated tools that can recommend settings specific to your server configuration that’s a one-click download.
- One of the biggest slow-down of Apache is server media files. But wait, I thought Free Republic doesn’t allow image uploading! Right, but Free Republic does server a few very important files. “/l/common.css” and “/favicon.ico”.
- Did you know the number one most requested file of any site is usually the Favicon.co file? It seems so innocent, but if optimized, can save hundreds of gigabytes in server bandwidth a month for high-trafficked websites such as FR. Currently the favicon is 32x32 at 2.2kb. The standard size is 16x16, which makes the favicon come out to 1.4kb. Even if we’re modest and suggest that Free Republic gets 1,000,000 unique views a month, that’s 781gb of data saved. With that file ALONE.
- While we’re on it, we can actually tell Apache to NOT server media files, but instead tell something that’s solid and powerful like memcached, a memory-mapping engine to server them instead. Once again, lowering the memory needed and increasing the number of concurrent connections.
- Anywhere on the site where FR has to query all users to return a result, it fails and crashes, that’s an immediate point of target for DDoSing. Does FR implement a hard firewall or an automated Firewall system that null route / blackholes attackers?
- Without changing a single thing in the site, just looking at what “slow” operations perl is doing could save milliseconds per page. Milliseconds over a few hundred people means people are getting pages faster in seconds. http://www.ibm.com/developerworks/library/l-optperl/index.html
- Changing how your browser downloads content is one solution, but the fault is on the server, not the user. Throwing money at the problem
- Install Newrelic for 1 hour to collect debugging information on the server. That’ll identify every problem you have and need fixed.
- I also suggest moving away from Apache, and use Lighttpd, as it uses less memory, less CPU time, and is significantly faster. Even Nginx is better.
- The good news is that your email hosting is on a different IP address than where FR is hosted, which is great.
Jim and John, you’ve got options. I’m sure this has all been mentioned before. There has never been a more critical time to pursue one or many of these options. You take pride in closing the world out to FR’s workings to protect it. That’s okay, but that means the future rests solely on your shoulders. You have an enormous amount of people who rely on you. It would be my suggestion to listen to their suggestions, something many suggest our current leaders aren’t doing today.
Best regards,
Tim Selaty Jr., Co-Founder of Tea Party Tribune
timselaty@gmail.com
Have eliminated the sidebar and limited posts to twenty. In the morning there are no problems with loading FR....BUT am completely unable to load, post etc about the time you come on the fund raising thread. Is it possible that someone has put a cookie or some negative load on those threads....Just an idea....
OTOH: Could the freepathon be turned off for tomorrows election? Would that help?
Another few suggestions:
1. Install “htop” to see where all the memory, CPU, and load averages is going. Once you’ve found a process you’d like to trace, simply his “s”.
2. Check to see where the memory in apache processes are coming from. Specifically, use “top” and “lsof” to see the top apache processes. Once you’ve found one that seems to be about average, install “lsof” and type “lsof -p {pid}”, where {pid} is the process ID of the apache process you want to diagnose.
3. I’ve had where my access log was literally being loaded into my apache processes each and every time. With high-trafficked websites, that’s wasted memory per process. You can quickly see if that’s a problem using “lsof”.
4. Switch FreeRepublic’s DNS to use CloudFlare. This way even if the site goes down, everyone has access to the site’s content and information. It’ll also help filter out bad and aggressive DDoS attempts. Less bad requests = faster FR.
Just a quick note to say hello and to thank Jim for all that you do, I’ve enjoyed FR for the past few years, and I’m looking forward to spending tomorrow here (fingers crossed) with everyone, may God bless us with a quick and decisive victory...
Just saw that you switched from Apache to Nginx. +1. Good luck tomorrow, FP.
bfl
if you could only come up with a B.S. filter!!!!
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.