The thread pool mechanism increases nginx performance by 9 times times

Original title: Thread Pools in NGINX Boost performance 9x!

Original Official address: https://www.nginx.com/blog/thread-pools-boost-performance-9x/

This article is a translation, non-literal translation.

First, the question
In general, Nginx is an event handler, a controller that gets connection events from the kernel and tells the system how to handle them.
In fact, when the operating system do read and write data scheduling, Nginx is working together with the system, so nginx can respond as quickly as possible.

Nginx-handled events can be time-out notifications, socket-readable or read-write notifications, or error notifications. After the nginx receives these messages, it will process them individually.
But all processes are done in a simple thread loop.
Nginx executes after removing an event from the message queue, such as an event to read and write to the socket. In most cases this is fast and the nginx is processed in an instant.

What happens if there is a long-time operation? The entire message-processing loop must wait for this lengthy operation to complete before it can continue processing other messages.

So, what we call a "blocking operation" actually means an operation that takes a long time to consume a message loop.
The operating system can be blocked for a variety of reasons, or wait for resource access, such as hard disks, mutexes, database synchronization operations, and so on.

For example, when Nginx wants to read a file that is not cached in memory, it is read from disk.
But the disk is slow, and even if other subsequent events do not require access to the disk, they also have to wait for the end of the access disk for this event.
As a result, latency increases and system resources are not fully utilized.

Some operating systems provide asynchronous read-write file interfaces, which can be used in Nginx (http://nginx.org/en/docs/http/ngx_http_core_module.html?&&&_ga= 1.197764335.1343221768.1436170723#aio).
For example, FreeBSD is a good example, but unfortunately, Linux provides a series of asynchronous read file interface has a lot of flaws.
One of the problems is that file access and buffering require a queue, but Nginx is well resolved.
But there is one more serious problem: Using an asynchronous interface requires setting the O_direct identity of the file descriptor, which means that any access to the file skips the cache directly to the files on the disk.
In most cases, this is not the best way to access files.

Second, thread pool
To solve this problem, Nginx 1.7.11 introduced the thread pool concept. Now let's take a look at how the thread pool works.

In Nginx, the thread pool executes the distribution service, which consists of a task queue and some threads that perform the task.
When a worker is performing a task that could potentially have a long operation, the task is "unloaded" and re-placed into the task queue, and the "unload" task may be executed by another thread.



Now, there are only 2 base operations that cause "unload tasks" to the task queue:
1. Read () system calls on most operating systems
Sendfile of the 2.linux system ()
If this mechanism proves to be beneficial to nginx, we will add other actions later.

Third, the thread pool is not a panacea

Most read-write file operations need to pass through a slow disk. If there is sufficient memory to store the data, the operating system caches frequently used files, the page cache mechanism.

Because of the page caching mechanism, Nginx can show very good performance in almost all cases.
Reading data through the page cache is very fast and does not block.
On the other hand, removing a task to a task pool is a bottleneck.
So when the memory is plentiful and the data used is not very large, nginx works almost at its best even if the thread pool is not used.

Removing writes to the task pool is a processing scenario for a special scenario that is suitable for a large number of request operations that cannot be cached using the VM.
For example, a high-load nginx-based video streaming server.

In addition, FreeBSD users do not need to worry about this, because FreeBSD has a good asynchronous read interface, no need to use the thread pool.

Iv. Configuring the thread pool
If you are sure that the thread pool is appropriate for your scenario, take a look at how to configure the threading pools.
The steps for preparing the work are as follows:
1. Use Nginx 1.7.11 or newer version
2. Compiling nginx using the--with-threads parameter

In the simplest case, add an AIO thread ID (which can be added to an HTTP, server, or location segment):

AIO threads;

This is one of the simplest configuration examples, equal to the following configuration:

 # in  the  " main     Contextthread_pool  default  threads=65536   in  the  '   server   ' , or  

The above configuration defines a thread pool called default, with 32 worker threads and a maximum of 65,536 tasks for the task queue.
If the task queue is full, Nginx discards the task and prints the following log:
thread pool "NAME" Queue overflow:n tasks waiting
When this log appears, it means that you can increase your task queue, Or your system will not be able to handle so many tasks.

So in summary, you can configure your number of threads, the task queue length, the thread pool name.
You can also set up multiple thread pools to use in different places:

inch ' Main ' Contextthread_pool One threads= max_queue=0; thread_pool, Threads=32  ; http {    server {        /one{            AIO threads= one;        }         /both{            AIO threads= both;        }    }    ...}

If the max_queue, that is, the task queue length is unspecified, the length defaults to 65536.
Max_queue can also be set to 0, so that the thread pool can handle only as many tasks as the number of threads, and no tasks are stored in the task queue.

Now suppose you have a server with 3 hard disks, and you want this server to be used as a cache proxy, which is a cache node for your CDN, and the cached data has exceeded the available memory.
The most important thing in this scenario is to improve the performance of disk reads and writes.

One scenario is to use RAID, and the other is to use Nginx:

# We assume this hard drives isMounted on one of these directories:#/mnt/disk1,/mnt/disk2, or/mnt/disk3#inchThe'Main'Contextthread_pool Pool_1 Threads= -; thread_pool pool_2 Threads= -; thread_pool Pool_3 Threads= -; http {proxy_cache_path/mnt/disk1 levels=1:2keys_zone=cache_1:256m max_size=1024G Use_temp_path=off; Proxy_cache_path/mnt/disk2 levels=1:2keys_zone=cache_2:256m max_size=1024G Use_temp_path=off; Proxy_cache_path/MNT/DISK3 levels=1:2keys_zone=cache_3:256m max_size=1024G Use_temp_path=off; Split_clients $request _uri $disk {33.3%1; 33.3%2; *3; } server {... location/{proxy_pass http://backend;Proxy_cache_key $request _uri;            Proxy_cache Cache_$disk; AIO Threads=Pool_$disk;        Sendfile on; }    }}

In the configuration, the Thread_pool directive defines a separate thread pool for each disk;
The Proxy_cache_path directive defines separate cache paths and parameters for each disk;
The Split_clients module is used for load balancing of multiple caches (that is, multiple disks), a solution that fits the usage scenario;
The Use_temp_path=off parameter in Proxy_cache_path allows Nginx to store temporary files to the cache directory, which avoids data copying between disks when the cache is updated.

The above example shows that the nginx can be flexibly adjusted according to its own hardware.
With fine-tuning, you can get your software, operating system, and hardware working together in the best possible state, using all your resources as much as you could.

Conclusion

The thread pool mechanism is a very good mechanism to solve the blocking problems caused by the large number of data cases,
Makes Nginx's performance reach a new height.
As mentioned earlier, there is a new interface that could implement the "remove" task mechanism without lossy performance.

Note: Some of the translations in the original
(1) Offloading translation for unloading, in fact, is to put a task into the task pool;
(2) It is mentioned in the original text that the task is offloading to the thread pool, but in fact it is the task that is stored in the task pool, so I translate it to "unload the task into the task pools";
(3) Performance Test stage translation is skipped, you can refer to the original posts

The thread pool mechanism increases nginx performance by 9 times times