Nginx multi-process connection request/Event distribution Process Analysis

Nginx uses a multi-process approach for task processing, with only one thread per worker process, and single-threaded loops to handle all monitored events. This paper focuses on the multi-process load balancing problem and Nginx multi-process event processing process, convenient for everyone to write their own program when reference.

First, the monitoring establishment process

The entire establishment of the monitoring socket to accept the process such as:

Description

1.main inside calls Ngx_init_cycle (src/core/ngx_cycle.c), ngx_init_cycle inside to complete a lot of basic configuration, such as files, shared memory, sockets and so on.

2. The upper-left corner is the Ngx_open_listening_sockets (SRC/CORE/NGX_CONNECTION.C) that is called inside the ngx_init_cycle, including the basic creation socket,setsockopt , bind and listen, and so on.

3. Then the normal child process generation process. In the ngx_worker_process_cycle of each child worker process, the initialization operation Init_process of each module is called in the call Ngx_worker_process_init. Epoll Module For example, here call Ngx_event_process_init, which initializes multiple ngx_event_module types of module.ngx_event_module type only Ngx_event_core _module and Ngx_epoll_module. The actions section of the previous module is empty. The init function inside the ngx_epoll_module is ngx_epoll_init. The Ngx_epoll_init function mainly completes the Epoll part related initialization, including epoll_create, setting ngx_event_actions and so on.

4. Initialize the Ngx_epoll_module, continue the ngx_event_process_init, and then cycle through the read handler for each listening socket to Ngx_event_ Accept. Finally, each listening socket's read event is added to Epoll for waiting.

After the 5.ngx_event_process_init initialization is complete, each worker process starts looping through the events&timers. The final call is epoll_wait. As a result of the previous listening socket and join to Epoll, so if the listener word has a read message, then long call rev->handler for processing, the listener word handler has been set to ngx_event_accept. The ngx_event_accept is primarily called the Accept function to receive a new client socket for the clients sockets.

Here is the processing function for the listener word ngx_event_accept Flowchart:

Description

1. The first half is to accept the new connection Word by accepting, generate and set the relevant structure, and then add it to Epoll.

2. The second half calls the listening corresponding handler in connection, which is ngx_xxx_init_connection, where xxx can be mail,http and stream. As the name implies, this function is mainly to do the initialization of the new accepted connection word. Take the HTTP module as an example, initialize the read handler that set the connection word, and so on.

Second, load balancing problem

In Nginx, a load balancing policy is implemented between processes to obtain client connection requests through a variable ngx_accept_disabled. Ngx_accept_disabled using Flowchart:

Description

1.ngx_process_events_and_timers function, the current process load is judged by the positive and negative of the ngx_accept_disabled (greater than 0, high load; less than 0, low load). If the low load, do not do processing, the process to apply for the accept lock, listen and accept the new connection.

2. If the load is high, the ngx_accept_disabled will work. At this point, do not apply for the accept lock, let out monitoring and accept the new connection opportunities. At the same time ngx_accept_disabled minus 1, indicating that the load of the process will be slightly reduced by giving up the chance of an accept request until the ngx_accept_disabled is finally less than 0, re-entering the low-load state, and starting a new accept lock competition.

Reference Link: http://www.jb51.net/article/52177.htm

Three, "surprise group" problem

"Surprise Group" problem: multiple processes simultaneously listen to a socket, when a new connection arrives, will wake up all processes at the same time, but only one process and client connection success, resulting in a waste of resources.

Nginx through the process of sharing mutual exclusion lock Ngx_accept_mutex to control multiple worker processes mutually exclusive access to the common listener socket, obtain the lock after the call accept out with the client has established a connection to join Epoll, and then release the mutex.

Nginx Processing Process:

Description

1.ngx_accept_disabled as a token of a high per-process load (7/8 of the maximum allowable connections), the formula is calculated:

ngx_accept_disabled = NGX_CYCLE->CONNECTION_N/8-ngx_cycle->free_connection_n;

That is, if the number of free_connection_n connections is less than 1/8 of the total number of connections Connection_n ngx_accept_disabled is greater than 0, otherwise less than 0. or ngx_accept_disabled less than 0 o'clock, Indicates that the number of available connections is higher, the load is low, ngx_accept_disabled is greater than 0 o'clock, the number of available connections is lower and the load is high.

2. If the process load is low, that is, ngx_accept_disabled is less than 0, the process is allowed to compete for the accept lock.

3. If the process load is high, the competitive accept lock is discarded and the ngx_accept_disabled minus 1, that is, the load is slightly reduced (ngx_accept_disabled less than 0 is available) due to the opportunity to concede a competitive accept lock. because the load is high (ngx_accept_disabled >0) just ngx_accept_disabled minus 1, here does not apply for accept lock, so the subsequent accept function will encounter "surprise group" problem, Return error Errno=eagain, return directly (personally feel there is room for improvement, see Supplemental section).

The Ngx_process_events_and_timers function section code is as follows:

1 if(Ngx_use_accept_mutex) {2         if(Ngx_accept_disabled >0) {3ngx_accept_disabled--;4 5}Else {6             if(Ngx_trylock_accept_mutex (cycle) = =ngx_error) {7                 return;8             }9 Ten             if(Ngx_accept_mutex_held) { OneFlags |=ngx_post_events; A  -}Else { -                 if(Timer = =Ngx_timer_infinite the|| Timer >ngx_accept_mutex_delay) -                 { -Timer =Ngx_accept_mutex_delay; -                 } +             } -         } +}

4. If the competition lock fails (6-7 rows), return directly, return to the ngx_worker_process_cycle for loop, this time not participate in event processing, the next cycle.

5. If the competition lock succeeds, the NGX_POST_EVENTS flag is set, which means that the event is placed in the queue, processed later, prioritized to release Ngx_accept_mutex, preventing a single process from taking too much lock time and affecting the efficiency of event processing. The Ngx_epoll_process_events function has the following part (same as the Write Event Wev section):

 1  if  (Flags & Ngx_post_events) { 2  queue = rev->accept?                         &ngx_posted_accept_events  3  : &ngx_posted_events;  4  //  first put the event in the queue and process it later  6  7 }  { 8  rev->handler (rev);  9 } 

6. Return the ngx_process_events_and_timers from Ngx_epoll_process_events, then handle the Accept event (line 10 below), after processing the Accept event, Release the lock immediately (line 13-15 below) and give other processes the opportunity to listen for connection events. Finally, the general connection event is handled.

1Delta =ngx_current_msec;2 3(void) ngx_process_events (cycle, timer, flags);4 5Delta = ngx_current_msec-Delta;6 7NGX_LOG_DEBUG1 (Ngx_log_debug_event, Cycle->log,0,8                    "Timer Delta:%M", delta);9 TenNgx_event_process_posted (cycle, &ngx_posted_accept_events);//This handles the accept event of post in Ngx_process_events One  A //release the lock immediately after processing the Accept event - if(Ngx_accept_mutex_held) { -Ngx_shmtx_unlock (&Ngx_accept_mutex); the } -  - //The timeout is processed before the normal connection event is processed.  - if(Delta) { + ngx_event_expire_timers (); - } +  A //handling a normal connection event request atNgx_event_process_posted (cycle, &ngx_posted_events);

7. When handling the Accept event, Handler is Ngx_event_accept (SRC/EVENT/NGX_EVENT_ACCEPT.C), in which a new connection for each accept is updated Ngx_accept_ Disabled

1  Do {2 ...3 //Accept new Connections4 accept ();5 ...6 //Update ngx_accept_disabled7ngx_accept_disabled = ngx_cycle->connection_n/88-Ngx_cycle->Free_connection_n;9 Ten ... One  A} while(ev->available)

Add:

ngx_accept_disabled minus 1 This path obviously did not apply for the accept lock, so the following epoll_wait and accept function will appear "surprise group" problem. Recommendations as improved:

Description

Add the red box step, when the load is too high, ngx_accept_disabled minus 1 for the equalization operation, and the Accept event is purged from the current process epoll. This epoll the current loop to handle only its own normal connection events. Of course, the left path may be executed several times, and the ngx_disable_accept_events operation needs to be performed only once.

If after a period of time, the process load down, into the right path, in the request to accept the lock function in Ngx_trylock_accept_mutex, after the application lock successful, will call Ngx_enable_accept_ Events adds the Accept event to the Epoll so that the accept event and the normal connection event can be monitored.

The above supplement is personal understanding, there are mistakes, please correct me.

Four, multi-process (single thread per process) reasons for efficiency

A little thought:

1.master/worker Multi-process mode to ensure the stability of the system. Master is more convenient for managing multiple worker sub-processes and other child processes. As the number of worker processes is the same as the number of CPU cores, there is no large number of child process generation and management tasks, which avoids the data IPC sharing overhead and switching contention overhead of a large number of child processes. Each worker process is also a duplicate copy of the listener word, in addition to the parent-child process to pass control messages, basically no IPC requirements.

2. Per worker single thread, there is no build and synchronization overhead for a large number of threads.

The above two aspects have made nginx avoid excessive synchronization, competition, switching and IPC data transfer, that is, as far as possible to free the CPU unnecessary computing overhead, only focus on business computing and process processing.

After freeing the CPU, it is the efficient operation of the memory. Like cache_manager_process, memory pool ngx_pool_t and so on. There is also the ability to set the affinity of the process to bind the CPU to a single core.

Such models are simpler, and larger connections are more scalable.

"The great things are always simple", the remark is not false.

Note: Refer to my article please indicate the source, thank you.

Nginx multi-process connection request/Event distribution Process Analysis