[Nginx] event-driven framework processing process

The ngx_event_core_module ngx_event_process_init method initializes the event module. This includes setting the handler corresponding to a read event such as "request connection" to the ngx_event_accept function and adding the event to the epoll module. When a new connection event occurs, ngx_event_accept is called. The general process is as follows:

1. The worker process continuously calls the ngx_process_events_and_timers function cyclically in the ngx_worker_process_cycle method to process events. This function is the total entry point for event processing.
2. Ngx_process_events_and_timers calls ngx_process_events. This is a macro, which is equivalent to ngx_event_actions.process_events. ngx_event_actions is a global struct and stores 10 function interfaces of the corresponding event Driver Module (epoll module. Therefore, the ngx_epoll_module_ctx.actions.process_events function is called to process events.
3. Ngx_epoll_process_events calls the Linux function interface epoll_wait to obtain the "New Connection" event, and then calls the handler processing function of the event to process the event.
4. As mentioned above, Handler has been set to the ngx_event_accept function, so you can call ngx_event_accept for actual processing.

The ngx_event_accept method is analyzed as follows:

The simplified code is as follows:

Using (ngx_event_t * eV) {socklen_t socklen; ngx_err_t err; ngx_log_t * log; using level; ngx_socket_t s; ngx_event_t * Rev, * WEV; using * ls; ngx_connection_t * C, * LC; ngx_event_conf_t * ECF; u_char SA [ngx_sockaddrlen]; If (ev-> timedout) {If (ngx_enable_accept_events (ngx_cycle_t *) ngx_cycle )! = Ngx_ OK) {return;} ev-> timedout = 0;} ECF = ngx_event_get_conf (ngx_cycle-> conf_ctx, ngx_event_core_module); If (ngx_event_flags & ngx_use_rtsig_event) {ev-> available = 1;} else if (! (Ngx_event_flags & ngx_use_kqueue_event) {ev-> available = ECF-> multi_accept;} lc = ev-> data; LS = LC-> listening; ev-> ready = 0; do {socklen = ngx_sockaddrlen;/* 1. The accept method tries to establish a connection, and the non-blocking call */S = accept (LC-> FD, (struct sockaddr *) SA, & socklen); If (S = (ngx_socket_t)-1) {err = ngx_socket_errno; If (ERR = ngx_eagain) {/* no connection, directly return */return ;} level = ngx_log_alert; If (ERR = ngx_econnaborted) {Level = ngx_log_err;} else if (ERR = ngx_emfile | err = ngx_enfile) {level = ngx_log_crit;} If (ERR = ngx_econnaborted) {If (ngx_event_flags & ngx_use_kqueue_event) {ev-> available --;} If (ev-> available) {continue;} If (ERR = ngx_emfile | err = ngx_enfile) {If (ngx_disable_accept_events (ngx_cycle_t *) ngx_cycle )! = Ngx_ OK) {return;} If (latency) {ngx_shmtx_unlock (& ngx_accept_mutex); latency = 0;} ngx_accept_disabled = 1;} else {ngx_add_timer (EV, ECF-> accept_mutex_delay);} return;}/* 2. Set the Server Load balancer threshold */ngx_accept_disabled = ngx_cycle-> connection_n/8-ngx_cycle-> free_connection_n; /* 3. Obtain a connection object from the connection pool */C = ngx_get_connection (S, ev-> log);/* 4. Create a memory pool */C-> pool = ngx_create_pool (LS-> pool_size, ev-> log); C-> sockaddr = ngx_palloc (c-> pool, socklen ); ngx_memcpy (c-> sockaddr, SA, socklen); log = ngx_palloc (c-> pool, sizeof (ngx_log_t )); /* set a blocking mode for AIO and non-blocking mode for others * // * 5. Set the socket attribute to blocking or non-blocking */If (ngx_inherited_nonblocking) {If (ngx_event_flags & ngx_use_aio_event) {If (ngx_blocking (S) =-1) {ngx_log_error (ngx _ Log_alert, ev-> log, ngx_socket_errno, ngx_blocking_n "failed"); ngx_close_accepted_connection (c); Return ;}} else {If (! (Ngx_event_flags & (blocks | blocks) {If (ngx_nonblocking (S) =-1) {ngx_log_error (ngx_log_alert, ev-> log, delimiter, ngx_nonblocking_n "failed "); ngx_close_accepted_connection (c); Return ;}}* log = LS-> log; C-> Recv = ngx_recv; C-> send = ngx_send; C-> recv_chain = ngx_recv_chain; c-> send_chain = ngx_send_chain; C-> log = log; C-> pool-> log = log; C-> socklen = socklen; C-> listening = ls; c-> local_sockaddr = LS-> sockaddr; C-> local_socklen = LS-> socklen; C-> unexpected_eof = 1; REV = C-> read; WEV = C-> write; WEV-> ready = 1; if (ngx_event_flags & (ngx_use_aio_event | ngx_use_rtsig_event) {/* rtsig, AIO, iocp */REV-> ready = 1;} If (ev-> deferred_accept) {rev-> ready = 1;} rev-> log = log; WEV-> log = log;/** todo: MT:-ngx_atomic_fetch_add () * or protection by critical section or light mutex ** todo: MP: -allocated in a shared memory *-ngx_atomic_fetch_add () * or protection by critical section or light mutex */C-> Number = ngx_atomic_fetch_add (ngx_connection_counter, 1 ); if (LS-> addr_ntop) {C-> addr_text.data = ngx_pnalloc (c-> pool, LS-> addr_text_max_len); If (c-> addr_text.data = NULL) {dependencies (c); return;} c-> addr_text.len = ngx_sock_ntop (c-> sockaddr, C-> socklen, C-> addr_text.data, LS-> addr_text_max_len, 0 ); if (c-> addr_text.len = 0) {ngx_close_accepted_connection (c); Return ;}} /* 6. Add the read/write events corresponding to the new connection to the epoll object */If (ngx_add_conn & (ngx_event_flags & ngx_use_epoll_event) = 0) {If (ngx_add_conn (c) = ngx_error) {ngx_close_accepted_connection (c); Return ;}} log-> DATA = NULL; log-> handler = NULL;/* 7. Method for successful TCP call, in the ngx_listening_t struct */LS-> handler (c);} while (ev-> available);/* available indicates that as many connections can be established at a time, determined by the configuration item multi_accept */}

Nginx generally runs multiple worker processes, which listen to the same port at the same time. When there is a new connection, the kernel will wake up all these processes, but only one process can be connected to the client successfully, causing other processes to waste a lot of overhead in wake-up, this is called the "Shocking group" phenomenon. The nginx solution to the "surprise group" is to allow the process to obtain the ngx_accept_mutex mutex and allow the process to access a certain critical section. In this critical section, the process adds the read event corresponding to the connection it listens to the epoll module, so that when a "New Connection" event occurs, the worker process will respond. The process of locking and adding events is completed in the ngx_trylock_accept_mutex function. When other processes enter the function to add read events, they find that the mutex lock is held by another process, so it can only return, and the events it listens to cannot be added to the epoll module, as a result, the system cannot respond to the "New Connection" event. But there will be a problem: when will the process holding the mutex lock release the mutex lock? It takes a long time to release the lock after it completes processing all the events. During this period, other worker processes cannot establish new connections, which is obviously not desirable. The nginx solution is to obtain the mutex through ngx_trylock_accept_mutex. After obtaining the ready read/write event and returning the event from epoll_wait, classify the events into the queue:

• Add the new connection event to the ngx_posted_accept_events queue.
• Put existing connection events into the ngx_posted_events queue

The Code is as follows:

If (flags & ngx_post_events) {/* handle this batch of events later */queue = (ngx_event_t **) (REV-> Accept? & Ngx_posted_accept_events: & ngx_posted_events);/* Add the event to the delayed execution queue */ngx_locked_post_event (Rev, queue);} else {rev-> handler (REV ); /* handle the event immediately without delay */}

Write events are processed similarly. The process then processes the events in the ngx_posted_accept_events queue. After processing, the mutex lock is released immediately, minimizing the lock occupation time of the process.
Load Balancing in nginx each process uses a ngx_accept_disabled threshold for Server Load balancer. It is initialized in Step 1: ngx_accept_disabled = ngx_cycle-> connection_n/8-ngx_cycle-> free_connection_n; the initial value is a negative number, and the absolute value of this negative number is 7/8 of the total number of connections. when the threshold is less than 0, the system responds to the new connection event normally. When the threshold is greater than 0, the system no longer responds to the new connection event, and reduces ngx_accept_disabled by 1. The Code is as follows:

if (ngx_accept_disabled > 0){        ngx_accept_disabled--;}else{    if (ngx_trylock_accept_mutex(cycle) == NGX_ERROR)    {        return;    }    ....}

This indicates that when the current number of connections of a process reaches 7/8 of the total number of connections that can be processed, the Server Load balancer mechanism is triggered and the process stops responding to new connections.
Reference: Understanding nginx P328-P334.