Nginx c30k Problem Solving method _nginx

Our download service recently suffered a c30k, leading to the near stagnation of Nginx's download service. Why, very simple, the server deployed abroad, well-known reasons, the SL machine room line instability, coupled with the export rate of different regions are very sharp jitter, in order to speed up the download, we release the restrictions, allowing users to use multi-threaded download tools. As a result, c10k problems arise naturally. Download files are not small, each user at least 4 threads, while downloading several pieces of material ... It's natural to have more than 30k of concurrent links.

More limited by the money on hand, unable to expand (in fact, to be rich or not to run abroad). Therefore, you must improve your stand-alone concurrency capability and throughput.

Our download service is a plack application written in Perl, typical psgi, download verification, real-time firewall, user download tracking, and so on, can not directly use static file distribution (in fact, Perl performance is very efficient, deployed in the Starman, compared to the implementation of PHP, is about 10 times times the latter (PHP-FPM).

Starman is a nice psgi Server that uses the traditional prefork mode. Even if it's efficient, but prefork really can't handle c10k effectively, I can't increase Starman's worker to hundreds of thousand. Previous articles have mentioned that evented io is a solution that can cope with c10k. So I used Twiggy to replace Starman. Twiggy is a PSGI Server based on AE (anyevent), a single process. In the low concurrency, the single process of the Twiggy QPS is weaker than the Starman, but to the high concurrency, Twiggy advantage appears. In the actual deployment, I started a number of Twiggy processes, respectively listening to separate ports, Nginx use upstream for load balancing. The throughput of 10 Twiggy has far exceeded 50 Starman worker. Twiggy the cost is also small, so can be very relieved to increase the Twiggy process.

Thanks to the PSGI interface specification, switching from Starman to Twiggy, the application does not need to make any changes. (Provided that there is no blocking IO operation within the program).

Another problem is that the server's io-wait is relatively high, after all, downloading this is Io-bound task.

Nginx supports the Linux Native AIO, so I'm thinking about using AIO to significantly reduce io-wait? Should there be a noticeable improvement in performance?

There are some information on the internet, boasting nginx AIO performance promotion, magic and so on. I am a little skeptical, because there is no test data comparison, are all parrot. In addition, most configurations are more or less problematic.

I use the CentOS, Nginx AIO to use, must be centos more than 5.5. Because only 5.5 of the kernel have AIO Backport,nginx and do not use Libaio.

In addition, Nginx's AIO was originally developed for FreeBSD, Linux can be used, but by the many limitations of the Linux AIO.

1. Direct IO must be used. This results in the inability to use the VM's disk cache.
2. Files can use AIO only if the size and directio_alignment define the integer times of the block size, and the parts that cannot be rounded are read in blocking mode before and after the file blocks. That's why we need output-buffer. The directio_alignment size depends on the file system you use, the default is 512, and for XFS, note that if you do not modify the XFS bsize, you need to adjust to the XFS default 4k.

The configuration I used is as follows:

Copy Code code as follows:

location/archive {
Internal
AIO on;
Directio 4k;
Directio_alignment 4k;
Output_buffers 1 128k;
}

When AIO is enabled, you can see that cache memory consumption drops rapidly in Vmstat, because using AIO must use Directio, which bypasses the VM's diskcache.

What is the actual performance, AIO must be fast? Even Igor is not sure.

From our own actual effect, AIO does not have a noticeable performance boost, and on the contrary, occasionally slightly increases the io-wait, because the DiskCache is not available, and if the file majority and directio_alignment are biased (especially when the breakpoint is resumed), Most of the file read locations outside the directio_alignment data boundary), this part of the data must use blocking IO read, and no disk cache, add io-wait also understandable.

Finally, the conclusion is that instead of using less reliable nginx AIO, it is better to open some nginx of the worker, re-use VM disk cache, when memory 100% utilization, Nginx static file distribution efficiency is higher than the AIO mode.

BTW, this practical use case has also confirmed my point of view, do not believe that the online without test data, most of the rumors are copy & paste legend, each said, in fact, most of them have not actually proved.