Nginx Source Analysis-Main process-multi-process cluster and process load balancing processing

Therewas a panic about the socket's accept before the Linux 2.6 release. The later version has solved the problem.

Surprise group refers to the phenomenon that when multiple processes / threads are waiting for the same resource, all processes / threads compete for resources whenever resources are available.

Nginx uses a multi-process mode. Assuming that the Linux system is the 2.6 version, when there is a client to connect to the Nginx server, Nginx N process will listen to the socket accept, if all n processes are the client socket connection is monitored, it will cause the competition of resources and even data confusion . we want to make sure that a link is handled on one of the nginx processes, including the accept and Read/write events.

The key points of nginx to solve the load balance of surprise group and process

1. Nginx n process will scramble for file lock, when only get the file lock process, can handle the event of accept .

2. The process of not getting the file lock can only handle the read event of the current connection object

3. When the total number of connection connections for a single process reaches the total of 7/8 , no new Accpet events are received.

4. If the process to get the lock can quickly finish Accpet, and not get the lock has been waiting (waiting for delay:ngx_accept_mutex_delay), easy to cause the process busy busy, empty very empty

Specific implementation of the Ngx_process_events_and_timers process event dispatcher

This method is the core function of the process implementation. Main functions: Event distribution, surprise group processing, simple load balancing.

Load Balancing:

1. When the event configuration is initialized, a global variable is set: ngx_accept_disabled = NGX_CYCLE->CONNECTION_N/8-ngx_cycle->free_connection_n;

2. When ngx_accept_disabled is positive, when connection reaches 7/8 of the total number of connections, the new connection accept event is no longer processed and only the read event of the current connection is processed

Surprise group Treatment:

1. Through the Ngx_trylock_accept_mutex scramble for file locks, to get the file lock, you can handle the Accept event.

2. Ngx_accept_mutex_held is a flag to get the lock, when the lock is taken, flags will be set to ngx_post_events, this flag will be in the event handler function ngx_ Process_events All events (accept and read) into the corresponding ngx_posted_accept_events and ngx_posted_events queues Deferred processing .

3. When you do not get the lock, call the event handler function ngx_process_events , you can explicitly read the event, so you can call the event Ev->handler method callback processing.

4. To get the lock process, the next priority is to handle the Accept event on the ngx_posted_accept_events queue, processing the function:ngx_event_process_posted

5. After the Accept event is processed, the file lock is released

6. Next process the Read event on the ngx_posted_events queue, processing function:ngx_event_process_posted

/** * Process Event Dispatcher */void ngx_process_events_and_timers (ngx_cycle_t *cycle) {ngx_uint_t flags;ngx_msec_t timer, delta;if (NGX _timer_resolution) {timer = Ngx_timer_infinite;flags = 0;} else {timer = Ngx_event_find_timer (); flags = ngx_update_time;# if (NGX_WIN32)/* Handle signals from master in case of network inactivity */if (timer = = Ngx_timer_infinite | | Timer > {timer = 500;} #endif}/** * Ngx_use_accept_mutex variable represents whether the accept mutex is used by default and can be switched off by Accept_mutex off; * The function of the accept mutex is to avoid the cluster, while achieving load balancing */if (Ngx_use_accept_mutex) {/** * ngx_accept_disabled = NGX_CYCLE->CONNECTION_N/8-Ngx_cycle->free_connection_ N * When connection reaches 7/8 of the total number of connections, the new connection accept event is no longer processed, only the read event of the current connection is processed * This is a relatively simple load balancing method */if (ngx_accept_disabled > 0) {ngx_ accept_disabled--;} else {/* get lock Failed */if (Ngx_trylock_accept_mutex (cycle) = = Ngx_error) {return;} /* Get the Lock */if (Ngx_accept_mutex_held) {/** * adds tag ngx_post_events to the flags as a parameter of the core function ngx_process_events of the processing time, All events in this function will be deferred for processing. * The Accept event is put into the ngx_posted_accept_events chainTable, * Epollin|epollout Ordinary events are placed in the ngx_posted_events linked list **/flags |= ngx_post_events;} else {/** * 1. Acquiring a lock failure means that neither the current worker process can attempt to grab the lock frequently, nor let it go through too long an event to grab the lock * 2. The timer_resolution time accuracy is turned on, and you need to let Ngx_process_ The change method waits at least ngx_accept_mutex_delay milliseconds to try to grab the lock * 3 when there is no new event. When there is no time accuracy, the timer is set to Ngx_accept_mutex_delay milliseconds * 4 If the time-out distance for the most recent timer event is now more than ngx_accept_mutex_delay milliseconds. You cannot allow the Ngx_process_change method to wait longer than ngx_accept_mutex_delay when there are no new events, which affects the entire load balancing mechanism * 5. If the process to get the lock can quickly finish Accpet, and not get the lock has been waiting, easy to cause the process busy busy, empty very empty */if (timer = = ngx_timer_infinite| | timer > Ngx_accept_mutex_ Delay) {timer = Ngx_accept_mutex_delay;}}}} Delta = ngx_current_msec;/** * Event Dispatch function * 1. When the lock is flags=ngx_post_events, the event is not processed directly, and the accept event is placed in the Ngx_posted_accept_events,read event to ngx_posted_events queue * 2. When the lock is not taken, all that is processed is the read event, and the callback function is processed directly * parameters: Timer-epoll_wait Timeout time (ngx_accept_mutex_delay-delay lock Event Ngx_timer_ infinite-normal epollwait wait event) */(void) ngx_process_events (cycle, timer, flags);d Elta = Ngx_current_msec-delta;ngx_log_ DEBUG1 (ngx_log_debug_event, Cycle->log, 0, "tImer Delta:%M ", delta);/** * 1. Ngx_posted_accept_events is an event queue, staging Epoll from the listener interface wait to the Accept event * 2. This method is to loop through the Accpet event */ngx_event_process_posted (cycle, &ngx_posted_accept_events) on the Accpet event queue;/** * If you get the lock, After the Accept event is processed, the lock */if (Ngx_accept_mutex_held) {Ngx_shmtx_unlock (&ngx_accept_mutex) is released;} if (delta) {ngx_event_expire_timers ();} /** * * *. Normal events will be stored in the ngx_posted_events queue. This method is to loop through the Read event queue on the Read event */ngx_event_process_posted (cycle, &ngx_posted_events);}

Ngx_trylock_accept_mutex Get Accept Lock

1. Ngx_accept_mutex_held is the global variable that gets the unique identity of the lock.

2. When you get the lock, call ngx_enable_accept_events and add the new connection to the event

3. If you do not get the lock, call ngx_disable_accept_events.

/** * Get accept Lock */ngx_int_t Ngx_trylock_accept_mutex (ngx_cycle_t *cycle) {/** * get lock */if (Ngx_shmtx_trylock (&ngx_ Accept_mutex) {ngx_log_debug0 (ngx_log_debug_event, Cycle->log, 0, "accept mutex locked");/* come in several times to determine if you have got the lock */if ( Ngx_accept_mutex_held && ngx_accept_events = = 0) {return NGX_OK;} /* Call ngx_enable_accept_events, turn on the Listen Accpet event */if (ngx_enable_accept_events (cycle) = = Ngx_error) {Ngx_shmtx_unlock ( &ngx_accept_mutex); return ngx_error;} ngx_accept_events = 0;ngx_accept_mutex_held = 1;return ngx_ok;} NGX_LOG_DEBUG1 (ngx_log_debug_event, Cycle->log, 0, "Accept Mutex lock failed:%ui", Ngx_accept_mutex_held);/** * Did not get the lock, but ngx_accept_mutex_held=1 */if (Ngx_accept_mutex_held) {/* did not get the lock, call ngx_disable_accept_events, delete Accpet event */if (Ngx_disable_accept_events (cycle, 0) = = ngx_error) {return ngx_error;} Ngx_accept_mutex_held = 0;} return NGX_OK;}

Ngx_enable_accept_events and Ngx_disable_accept_events

/** * Open the Listen on the Accept event and add the Accept event to the event */static ngx_int_t ngx_enable_accept_events (ngx_cycle_t *cycle) {ngx_uint_t i; ngx_listening_t *ls;ngx_connection_t *c;ls = cycle->listening.elts;for (i = 0; i < cycle->listening.nelts; i++) { c = ls[i].connection;if (c = = NULL | | c->read->active) {continue;} if (Ngx_add_event (C->read, ngx_read_event, 0) = = ngx_error) {return ngx_error;}} return NGX_OK;} /** * Closes the listen for accept event * and removes the Accept event from the event */static ngx_int_t ngx_disable_accept_events (ngx_cycle_t *cycle, ngx_uint_t All) {ngx_uint_t i;ngx_listening_t *ls;ngx_connection_t *c;ls = cycle->listening.elts;for (i = 0; i < cycle->list Ening.nelts; i++) {c = ls[i].connection;/* if c->read->active, then is an active connection, already in use */if (c = = NULL | |!c->read->active) {Contin UE;} #if (ngx_have_reuseport)/* * Do not disable accept on worker's own sockets * when disabling accept events due to accept Mu Tex */if (Ls[i].reuseport &&!all) {continue;} #endif/* Delete event */if (Ngx_del_event (C->read, ngx_read_event,ngx_disable_event) = = Ngx_error) {return ngx_error;}} return NGX_OK;}

Ngx_event_process_posted Event Queue Processing

Callback handling of Accept/read events on the ngx_posted_accept_events or ngx_posted_events queues.

/** * Processing Event queue * */void ngx_event_process_posted (ngx_cycle_t *cycle, ngx_queue_t *posted) {ngx_queue_t *q;ngx_event_t *ev;w Hile (!ngx_queue_empty (posted)) {q = Ngx_queue_head (posted); ev = Ngx_queue_data (q, ngx_event_t, queue); NGX_LOG_DEBUG1 ( Ngx_log_debug_event, Cycle->log, 0, "posted event%p", Ev), ngx_delete_posted_event (EV);/* Event callback function */ev->handler ( EV);}}

Core processing functions of the Ngx_process_events event

This method, we mainly look at the ngx_epoll_process_events method under the Epoll model (NGX_EPOLL_MODULE.C)

1. If a lock is reached, the Accpet/read event is placed on the queue for deferred processing.

2. Processes that do not grab locks are processed directly by the Read event that handles the current connection.

        /* Read Event Epollin */        if ((Revents & Epollin) && rev->active) {#if (ngx_have_epollrdhup)            if (Revents & Amp Epollrdhup) {                rev->pending_eof = 1;            }            rev->available = 1; #endif            rev->ready = 1;            /* If the event grabs the lock, put the event queue *            /if (Flags & ngx_post_events) {                queue = rev->accept? &ngx_posted_accept_events< c10/>: &ngx_posted_events;                Ngx_post_event (rev, queue);            } else {            /* does not grab the lock, the read event is processed directly *                /Rev->handler (rev);}        }

At the beginning of the next section, we will enter the Nginx event module.

Nginx source Code Analysis-Main process-multi-process cluster and process load balancing processing