Search:  
Gentoo Wiki

Streamlining_your_web_server

This howto is provided to help people setup their server to better withstand high amounts of traffic. most notably a digg or a slashdot.

This document is provided as a look into and a history of what changes are made to the gentoo-wiki server to keep it operational even under high amounts of traffic.

Contents

Apache

HostNameLookups

HostnameLookups         off

This is SO very important, having this option on is one of the worst things you can have on an apache server. What it does is look up the fully qualified hostname of every client who accesses your web server; so instead of 10.1.5.67 in your apache logs you get something like bill.someserver.net. The only real use of this directive is to get country that the host came from, this ties up a lot of apache's time and is basically worthless.

NOTE: As of Apache 2.0.58, this variable is set to 'off' by default.

KeepAlive

KeepAlive               Off

The keep alive setting allows hosts that connect to your site to continue to request through a single child; although this keeps a client requesting all his/her files through a single child and sounds like a good idea, it isn't in practice. Newer browsers such as Firefox will request multiple files (usually images) at the same time using up multiple child processes at once, very few browsers will use a single stream anymore. If KeepAlive is on, each stream will remain open for a time set in your httpd.conf (10 seconds by default), and can keep many apache processes doing ultimately nothing. For these reasons, turn it off.

Child Servers

StartServers            25
MinSpareServers         10
MaxSpareServers         15
MaxClients              150
MaxRequestsPerChild     3000

These are the setting for gentoo-wiki and its child servers. By default MaxRequestsPerChild will either be 0 (infinite) or somewhere around 10 depending on which version of apache you installed. The MaxRequestsPerChild directive is THE most important out of this set, if its too small the apache processes will continually shut down and start up causing excess load on the server, too high and a memory leak could kill your server; we have chosen 3000 through trial and error as a good number to set this at. If your site has more static content (images) set the number higher, if it has less set the number lower; non-static content is more likely to cause a memory leak.

As for the server count this is at your discretion, we choose 150 because it best fits how many processes the server can handle without crashing.

Logging

TURN OFF LOGGING

Need I say more?, if your server is anything like mine it has a slow HD, and unless your HD is a solid state drive or ram disk yours is slow as-well. logging takes up a LOT of disk i/o and can slow your server to a crawl when you start to get a high amount of traffic, on our server turning off logging reduced the server load from 4.10 to 0.50 and reduced the open apache2 servers by half.

Now without logging you wont have stats with awstats or webalizer, but this cant be helped. If you want to be able to handle a slashdot or digg you need to turn off logging. Gentoo-wiki is now using mint for its logging, and there are many other javascript based loggers (google makes one) try using one of those (good open one is phpMyVisites).

Simply comment out any lines that specify logging to a file:

CustomLog logs/access_log common
ErrorLog logs/error_log

Site Design

Avoid dynamic pages wherever possible. At least your main pages and articles should be static or use "funky caching". Gentoo-wiki is based off mediawiki and makes use of memcached to cache pages.

PHP

PHP Accelerator

Gentoo-wiki uses PECL-apc as an accelerator.

Opcode Cachers

An Opcode Cacher stores 'Opcode' versions of PHP code reduces time for code execution. In varying percentages of improvement depending up code style and the Cacher involved. Generally XCache the best.

Variable Cachers

APC & XCache both provide *_get and *_set functionality. (These only function on one node, for a multi-node variable cache take a look at ebuild:net-misc/memcached.)

mySQL

Query Caching

set-variable    = query_cache_size=200M

setting this variable in your my.cf will cause your most common queries to be cached and stored for faster retrieval. Gentoo-wiki currently gets about a 96% cache hit rate which basically means that 96% of the queries requested will not actually be processed but you will simply get your result instantly. This is a great boost in performance and can shave a few milliseconds off every page.

the Gentoo-wiki server has a lot of ram which is why we have a 200M cache (only 50MB ever seems to get used thou) even a 20M cache will do wonders for your server

A good tool for manually tweaking your MySQL server settings is MySQL Administrator, available for Windows and Linux at mysql.com/products/administrator. - W

Squid

HTTPD Accelerator

Setting up an httpd accelerator with squid helps to keep apache processes from running, and runs squid processes to pass the data on to the client. Apache processes can have a huge footprint. also squid will cache commonly used images, css and javascript files in RAM making them serve faster with less disk access (which is slow).

Other

robots.txt

User-agent: Fasterfox
Disallow: /

This is just a small thing but will help you to drop some unwanted traffic from fasterfox and shouldn't really cause any noticeable difference to the end user.

Notes

These are the settings that are in use by Gentoo-wiki and are improved/tweaked as needed. If you feel you know a way to improve our servers performance please add a note to the discussion page.

Retrieved from "http://www.gentoo-wiki.info/Streamlining_your_web_server"

Last modified: Fri, 05 Sep 2008 02:53:00 +0000 Hits: 15,103