About nginx + FastCGI + Django

Recently, Django was used to develop a set of advertising systems, which are actually a set of online advertising alliance systems, including advertising, management, statistics, anti-cheating, and tracking. The hardware is a Dell PC Server (dual-core xeon1, 2 GB memory), the operating system is RedHat as4, and the others are nginx 0.5.33, Python 2.5, MySQL 5.0.41, django was obtained from SVN trunk around March.

At the beginning, the system was running under Apache + mod_python. Later, due to load problems, the nginx + FastCGI + Django mode was changed. Specifically, Django FastCGI runs in prefork mode, maxchildren = 10, and nginx connects to the FastCGI process through TCP socket. During a period of time, the system was running well and the load average was below 1. At that time, the advertisement clicks and display of the whole system was around 300 million/day.

Soon the traffic went up. When several major sites joined the Alliance, the system load immediately went up. Not counting static files, it almost reached 2000 million/day. Even worse, 502 gateway errors frequently occur. I tried to add maxchildren, but because the memory of this machine is not very large, it only has 2 GB. The half was allocated to MySQL, and some statistics and analysis programs were also run on this server, maxchildren cannot be too large. I was in a hurry. I switched Django FastCGI to threaded mode and adjusted some parameters! Here is a summary:

1. Django FastCGI runs in threaded mode, which saves resources than the prefork Mode

2. Previously, the threaded mode was not stable, but it has improved. For details, see
Http://www.pkblogs.com/timchen119/2007/06/thread-dying-problem-may-fixed-in.html

3. In Django FastCGI mode, the maxchildren parameter determines how many requests can be processed simultaneously by your service (except for static files processed by nginx ). If the number of requests exceeds the maxchildren setting, the error 502 may occur. The default value of maxchildren is 50.

4. Enable the nginx Status Module to monitor some nginx statuses, such:
Active connections: 5620
Server accepts handled requests
882256 882256 1300056
Reading: 35 writing: 133 waiting: 5452

Active connections indicates the number of active connections, which should be equal to the sum of three in the fourth row. In the three numbers in the fourth row, reading indicates the number of links that are reading data from the client, writing indicates the number of links that are returning data to the client. Waiting is usually because of keepalive, the number of links waiting for the next request. The number of reading + writing = the number of requests processed by dajngo + static file requests processed by nginx. You can set the number of maxchildren Based on reading + writing.

5. If Django runs in threaded mode and has a high load, it may not be enough to start a FastCGI process. One is unstable, and the other is Gil restrictions. Therefore, multiple FastCGI processes running in threaded mode may be required. My current temporary practice is to use an nginx virtual host for reverse proxy, followed by multiple groups of nginx + Django threaded FastCGI. You can try to adjust it later.

6. nginx communicates with FastCGI through UNIX socket, which is more efficient than TCP socket, and can be considered under heavy load.

For some new ideas about the prefork and threaded modes recently, please refer to the http://blog.csdn.net/FeiSan/archive/2008/01/23/2061208.aspx