Nginx Source Analysis-Nginx startup and IOCP model

nginx Source Analysis-Nginx startup and IOCP model version and Platform information

This document analyzes the related code under Windows for the Nginx1.11.7 version, although the server may be more Linux, but the code under the Windows platform is basically similar
, in addition to the IOCP completion port of Windows, the asynchronous IO model is excellent and well worth a look. nginx Boot

Once a friend asked me, the face of a large project source code, should read from what. I gave him an example, our school freshman sophomore is in the Zijin Port campus, to the
The junior move to Yuquan campus, but the freshman time will sometimes have things to do in Yuquan. Occasionally will go to Yuquan, but Yuquan campus unfamiliar, so follow Baidu map or
Follow the Seniors Walk. Because it is to do things, so generally is part of the walk, such as in the college building inside walk. When the junior first came to Yuquan, they would find that even their own
have been to several times before, also feel this campus completely unfamiliar, even to come to place, also appear strange. But when we really started learning about life at the Yuquan campus,
Every day from the bedroom walk to the classroom is a road, teaching super is another road, the two main road after several times, sometimes on the way to the side of the path to see, so slowly
Also familiar with this new campus.

Source code reading is not so, if there is not a major route, always take a partial look, not to mention, it is not easy to grasp the overall structure. Between the modules
's dependence is also easy to clarify. If there is a trunk line, read the source code, the overall structure and ideas will become clear. Of course, I also hold this view: blog,
The author of the article, the author of the idea of writing the article is clear, the reader is not necessarily able to see, and everyone writes things will inevitably be overlooked. Read the source code analysis guidelines written by others
And so on, in a more extreme words, is a kind of self-satisfaction, feel that they quickly learned a lot of source-level knowledge, but actually think, learning, more important is
Learning ability of exercise, through the source of learning, the process of learning their own situation in the thinking, even combined with the thinking of social philosophy, and read the source of income,
Oneself in peacetime use frame, library, out of the problem of the solution, flip through the source of others to find the ability of the bug. If you just read the source code analysis written by others, and write
Code only to copy the ready-made code, to some extent there is a certain similarity.

I am using go-based, before the first-class Nginx middleware is not much understanding, but also the first time to see, the level of deficiencies, but also look haihan. Back to the Chase,
Nginx source code Analysis, but also to find a major route, for many programs, the startup process is a very good route, find Nginx entrance function main
, found in /src/core/nginx.c , the code is probably as follows:

int ngx_cdecl Main (int argc, char *const *argv) {
    ...//First some variable declarations

    ngx_debug_init (); 
    ...
    Ngx_pid = Ngx_getpid ();
    ...
    Init_cycle.pool = Ngx_create_pool (1024x768, log);
    ...
    Cycle = Ngx_init_cycle (&init_cycle);
    ...
    if (ngx_signal) {
        return ngx_signal_process (cycle, ngx_signal);
    }
    ...
    if (Ngx_create_pidfile (&ccf->pid, cycle->log)! = NGX_OK) {
        return 1;
    }
    ...
    if (ngx_process = = ngx_process_single) {
        ngx_single_process_cycle (cycle);

    } else {
        ngx_master_process _cycle (cycle);
    }

    return 0
}

This code looks roughly, first doing something initialized, including assigning the pool to a variable that looks like memory pools, getting system information,
Initialize the log system and so on, because it has not entered the corresponding function to look closely, so first put. The students who used nginx should know that Nginx command line
Run./nginx, he ran the service directly, very quiet, and then use CTRL + C also shut down. But open another console, run
./nginx-h will see:

Nginx version:nginx/1.11.7
Usage:nginx [-?HVVTTQ] [-s signal] [-c filename] [-p prefix] [-g directives]

Options:
  -?,-H: This help-
  v            : Show version and exit-
  v            : Show version and configure options then Exit
  -T            : Test configuration and Exit            -T: Test configuration, dump it and exit-
  Q            : Suppress Non-err or messages during configuration testing-
  s signal     : Send signal to a master process:stop, quit, reopen, reload -
  P prefix     : Set prefix path (default:none)-
  c filename   : Set configuration file (Default:conf/nginx . conf)-
  G Directives:set Global directives out of configuration file

This is the Nginx command line parameter introduction, to exit Nginx need to use nginx-s stop to the already open Nginx process to send a signal, let it exit.
And Nginx also supports a smooth restart, which is very useful in changing the nginx configuration, the process of restarting the server, is actually nginx own internal
A process that re-loads the new configuration, but does not affect some of the existing connections, so it is called a smooth restart.

Gracefully Stop Nginx ...

and the main function after initialization, do is the command line parameter parsing, if the display version, then display a version of information, the exit;
Configuration file, then go to invoke the corresponding processing of the settings configuration file, if it is to send the control signal, then return ngx_signal_process (cycle, ngx_signal);
Processing signals and so on. Here is also a small trick, is about the PID file, the program put its own PID write a file, and then can prevent the start of multiple processes,
This is a relatively common trick. About Ngx_single_process_cycle (cycle) This should be a single-process scenario, typically the current server
Are multicore-based, so let's take a look at the main function of the Ngx_master_process_cycle master process. Main Process

ngx_master_process_cycle function in /src/os/win32/ngx_process_cycle.c , the function takes a parameter, which compares
Complex, but it can be seen that it should be related to the life cycle of each nginx cycle, it is considered that the nginx each smooth restart, is a cycle. Code
Divided into several parts:

void Ngx_master_process_cycle (ngx_cycle_t *cycle) {... if (ngx_process = = Ngx_process_worker) {//Ngx_p
        Rocess identifies the identity of the process, and if this process should be a worker process, go to the ngx_worker_process_cycle (cycle, ngx_master_process_event_name) that the worker should do;
    Return
    }
    ... Setenvironmentvariable ("Ngx_unique", Ngx_unique);

    Set the environment variable to indicate that the Nginx main process has already run ... ngx_master_process_event = CreateEvent (NULL, 1, 0, ngx_master_process_event_name);
                      if (ngx_master_process_event = = NULL) {ngx_log_error (Ngx_log_alert, Cycle->log, Ngx_errno,
        "CreateEvent (\"%s\ ") failed", Ngx_master_process_event_name);
    Exit (2);
    } if (Ngx_create_signal_events (cycle)! = Ngx_ok) {exit (2);

    } ngx_sprintf ((U_char *) ngx_cache_manager_mutex_name, "Ngx_cache_manager_mutex_%s%z", Ngx_unique); Ngx_cache_manager_mutex = CreateMutex (NULL, 0, Ngx_cache_manager_mutex_name
    );if (Ngx_cache_manager_mutex = = NULL) {ngx_log_error (Ngx_log_alert, Cycle->log, Ngx_errno, "
        CreateMutex (\ "%s\") failed ", Ngx_cache_manager_mutex_name);
    Exit (2);
    } Events[0] = ngx_stop_event;
    EVENTS[1] = ngx_quit_event;
    EVENTS[2] = ngx_reopen_event;

    EVENTS[3] = ngx_reload_event;

    Ngx_close_listening_sockets (cycle);
    if (Ngx_start_worker_processes (cycle, ngx_process_respawn) = = 0) {exit (2); }

    ...
}

Understanding this code requires an understanding of the little event-related api,createevent of the Windows system. You can create an event that can be followed by some methods
For example, SetEvent can cause this event to be activated, and the process or thread can also wait for an event to be sent through APIs such as WAITFORSINGLEOBEJCT.
Health This code is the creation of a number of events, including Stop,quit,reopen and reload, which are in ngx_create_signal_events
Created in the function:

Static ngx_int_t
ngx_create_signal_events (ngx_cycle_t *cycle)
{
    ngx_sprintf ((U_char *) Ngx_stop_event_ Name,
                "Global\\ngx_stop_%s%z", ngx_unique);

    Ngx_stop_event = CreateEvent (NULL, 1, 0, ngx_stop_event_name);
    if (ngx_stop_event = = NULL) {
        ngx_log_error (Ngx_log_alert, Cycle->log, Ngx_errno,
                      "CreateEvent (\"%s\ ") Failed ", ngx_stop_event_name);
        return ngx_error;
    }


    ngx_sprintf ((U_char *) ngx_quit_event_name,
                "Global\\ngx_quit_%s%z", ngx_unique);

...
}

After that, the main process calls Ngx_close_listening_sockets (cycle) to close the socket that is listening so that subsequent connections will not come in.
Because the main process loop is definitely called when it is restarted or initialized. Then call the Ngx_start_worker_processes function to start the worker
Thread. Let's look at the ngx_start_worker_process function, also in this file:

Static ngx_int_t
ngx_start_worker_processes (ngx_cycle_t *cycle, ngx_int_t type)
{
    ngx_int_t         n;
    ngx_core_conf_t  *CCF;

    Ngx_log_error (Ngx_log_notice, Cycle->log, 0, "Start worker Processes");

    CCF = (ngx_core_conf_t *) ngx_get_conf (cycle->conf_ctx, ngx_core_module);

    for (n = 0; n < ccf->worker_processes; n++) {
        if (ngx_spawn_process (cycle, "worker", type) = = Ngx_invalid_pid) { Break
            ;
        }
    }

    return n;
}

This function first reads the configuration of this loop, and starts the corresponding number of worker processes according to the settings of the worker_process in the configuration file
In the/conf/nginx.conf:

#user  Nobody;
Worker_processes  8;

#error_log  Logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


Events {
    worker_connections  65536;
}
...

Of course, if there are no settings in the configuration file, and how to set the default values in the newly created configuration file, these are all in/src/core/nginx.c, but are not
Very important, so temporarily skipped. The regression ngx_master_process_cycle function, which, after creating an event, enters a dead loop:

 for (;;)
        {Nev = 4; for (n = 0; n < ngx_last_process; n++) {if (Ngx_processes[n].handle) {events[nev++] = Ngx
            _processes[n].handle;
        }} if (timer) {timeout = timer > ngx_current_msec? timer-ngx_current_msec:0;

        } EV = WaitForMultipleObjects (Nev, events, 0, timeout);
        err = Ngx_errno;

        Ngx_time_update ();

        NGX_LOG_DEBUG1 (Ngx_log_debug_core, Cycle->log, 0, "Master waitformultipleobjects:%ul", Ev);

            if (ev = = wait_object_0) {ngx_log_error (ngx_log_notice, Cycle->log, 0, "exiting");
                              if (resetevent (ngx_stop_event) = = 0) {ngx_log_error (Ngx_log_alert, Cycle->log, 0,
            "ResetEvent (\"%s\ ") failed", Ngx_stop_event_name);
            } if (timer = = 0) {timer = ngx_current_msec + 5000; } Ngx_termiNate = 1;

            Ngx_quit_worker_processes (cycle, 0);
        Continue

            } if (ev = = wait_object_0 + 1) {Ngx_log_error (ngx_log_notice, Cycle->log, 0, "shutting down");
                              if (resetevent (ngx_quit_event) = = 0) {ngx_log_error (Ngx_log_alert, Cycle->log, 0,
            "ResetEvent (\"%s\ ") failed", Ngx_quit_event_name);
            } ngx_quit = 1;

            Ngx_quit_worker_processes (cycle, 0);
        Continue } ... if (Ev > WAIT_OBJECT_0 + 3 && EV < WAIT_OBJECT_0 + Nev) {ngx_log_debug0 (

            Ngx_log_debug_core, Cycle->log, 0, "reap worker");

            Live = Ngx_reap_worker (cycle, Events[ev]);
            if (!live && (ngx_terminate | | ngx_quit)) {ngx_master_process_exit (cycle);
        } continue; } if (ev = = wait_timeout) {ngx_terminate_worker_processes (CycLe);
        Ngx_master_process_exit (cycle);
                          } if (ev = = wait_failed) {ngx_log_error (Ngx_log_alert, Cycle->log, err,

            "WaitForMultipleObjects () failed");
        Continue } ngx_log_error (Ngx_log_alert, Cycle->log, 0, WaitForMultipleObjects () returned unexpected value%
    UL ", Ev); }

This first introduces WaitForMultipleObjects, which waits for multiple kernel objects, which can be events, locks, processes, and so on. In this cycle, each time
The loop first adds ngx_last_process processes to the event array, which is probably the process group used in the last loop. If the defined
Stop,quit,reload,reopen four kinds of event triggering, call the relevant function to close or restart the worker process, respectively. If this is the last loop used in the
Die, then go to restart the process, call the Ngx_reap_worker function, this function after confirming that the old process has died, will call Ngx_spawn_process
To restart a new process. Ngx_spawn_process will call Ngx_execute to open a new process, this part of the details, put in the next section to talk about. So we
Knowing that the main process starts, it goes into the event processing loop to handle the instructions sent by nginx-s and the restart that handles the death of the process group. So let's look at the worker process
What to do. worker Process

We understand that the main process calls the Ngx_start_worker_process function to start multiple worker processes based on the configuration file, which calls the Ngx_spawn_process
To start a new worker process, let's take a look at how ngx_spawn_process starts a new process. The following is a partial code (located in /src/os/win32/ngx_process.c):

ngx_pid_t ngx_spawn_process (ngx_cycle_t *cycle, Char *name, ngx_int_t respawn) {...//variable definition//First time main loop incoming Ngx_pro

    cess_just_respawn==-3 if (respawn >= 0) {s = respawn;
                } else {for (s = 0; s < ngx_last_process; s++) {if (Ngx_processes[s].handle = = NULL) {
            Break
                          }} if (s = = ngx_max_processes) {ngx_log_error (Ngx_log_alert, Cycle->log, 0,
            "No more than%d processes can be spawned", ngx_max_processes);
        return ngx_invalid_pid;

    }}//get Nginx file path n = GetModuleFileName (NULL, file, MAX_PATH); if (n = = 0) {ngx_log_error (Ngx_log_alert, Cycle->log, Ngx_errno, "GetModuleFileName () FA
        Iled ");
    return ngx_invalid_pid;
    } File[n] = ' + ';
    ... ctx.path = file; ... pid = Ngx_execute (cycle, &ctx); Create New process ...

This part is to find the index in the ngx_process, and then put in a new process, then we see how the Ngx_execute function is executed:

ngx_pid_t Ngx_execute (ngx_cycle_t *cycle, ngx_exec_ctx_t *ctx)
{
    ...
    if (CreateProcess (Ctx->path, Ctx->args,
                      null, NULL, 0, Create_no_window, NULL, NULL, &SI, &pi)
        = = 0)
    {
        ngx_log_error (Ngx_log_crit, Cycle->log, Ngx_errno,
                      CreateProcess (\ "%s\") failed ", Ngx_argv[0] );

        return 0;
    }

    Ctx->child = pi.hprocess;

    if (CloseHandle (pi.hthread) = = 0) {
        ngx_log_error (Ngx_log_alert, Cycle->log, Ngx_errno,
                      "CloseHandle ( Pi.hthread) failed ");
    }

    Ngx_log_error (Ngx_log_notice, Cycle->log, 0,
                  "start%s process%P", ctx->name, Pi.dwprocessid);

    return pi.dwprocessid;
}

This function opens a new process by calling Ngx_execute and stores the process handle in the context and returns the PID. After the system process is created, a call to the
WaitForMultipleObjects waits for two events, one is Ngx_master_process_event, the event is defined in the main process loop, and the other
One is the new process of death. If the main process event is triggered, the event for the new process is set using OpenEvent for the previously created event. But maybe it's because
The version I have in hand is still under development, and I have not found a statement about this manual event trigger in the code. Another event is the death of the new process, if the event
Is triggered, some cleanup code (kill process, etc.) is executed.

But we found that the CreateProcess inside is just a new launch of the Nginx, then the new start of the nginx process will become a worker process.
Remember that there are initialization os_init for the operating system in the main function, and this function has some code in the implementation of Win32 as follows:

ngx_int_t
ngx_os_init (ngx_log_t *log)
{
    ...
    if (GetEnvironmentVariable ("Ngx_unique", Ngx_unique, Ngx_int32_len + 1)!
        = 0)
    {
        ngx_process = Ngx_process_ WORKER;

    }
    ...

And ngx_process is the condition that determines the cycle of a master process changing to a worker process:

void Ngx_master_process_cycle (ngx_cycle_t *cycle) {
    ...
    if (ngx_process = = Ngx_process_worker) { 
        ///ngx_process identifies each process's identity, and if this process should be a worker process, then the worker should do the
        Ngx_worker_ Process_cycle (cycle, ngx_master_process_event_name);
        return;
    }
    ...
    Setenvironmentvariable ("Ngx_unique", Ngx_unique); Set the environment variable to indicate that the Nginx master process is already running

So let's take a look at the main loop for this worker process:

static void Ngx_worker_process_cycle (ngx_cycle_t *cycle, char *mevn) {...//variable definition log = cycle->log;

    NGX_LOG_DEBUG0 (ngx_log_debug_core, log, 0, "worker started");
    ngx_sprintf ((U_char *) WTEVN, "Ngx_worker_term_%p%z", ngx_pid);
    Events[0] = CreateEvent (NULL, 1, 0, WTEVN); if (events[0] = = NULL) {ngx_log_error (Ngx_log_alert, log, Ngx_errno, "CreateEvent (\"%s\ ") F
        Ailed ", WTEVN);
    Goto failed;
    } ngx_sprintf ((U_char *) WQEVN, "Ngx_worker_quit_%p%z", ngx_pid);
    EVENTS[1] = CreateEvent (NULL, 1, 0, WQEVN); if (events[1] = = NULL) {ngx_log_error (Ngx_log_alert, log, Ngx_errno, "CreateEvent (\"%s\ ") F
        Ailed ", WQEVN);
    Goto failed;
    } ngx_sprintf ((U_char *) WROEVN, "Ngx_worker_reopen_%p%z", ngx_pid);
    EVENTS[2] = CreateEvent (NULL, 1, 0, WROEVN); if (events[2] = = NULL) {ngx_log_error (Ngx_log_alert, log, Ngx_errno, "CreateEvent (\"%s\ ") F AiLed ", WROEVN);
    Goto failed;
    } MEV = openevent (event_modify_state, 0, MEVN); if (MeV = = NULL) {ngx_log_error (Ngx_log_alert, log, Ngx_errno, "OpenEvent (\"%s\ ") failed",
        MEVN);
    Goto failed; } if (SetEvent (MEV) = = 0) {ngx_log_error (Ngx_log_alert, log, Ngx_errno, "SetEvent (\"%s)
        \ ") failed", MEVN);
    Goto failed;

    } ngx_sprintf ((U_char *) ngx_cache_manager_mutex_name, "Ngx_cache_manager_mutex_%s%z", Ngx_unique); Ngx_cache_manager_mutex = OpenMutex (SYNCHRONIZE, 0, Ngx_cache_manager_mutex_n
    AME); if (Ngx_cache_manager_mutex = = NULL) {ngx_log_error (Ngx_log_alert, log, Ngx_errno, "Openmut
        EX (\ "%s\") failed ", Ngx_cache_manager_mutex_name);
    Goto failed;
    } ngx_cache_manager_event = CreateEvent (null, 1, 0, NULL); if (ngx_cache_manager_event = = NULL) {ngx_log_error (NGx_log_alert, Cycle->log, Ngx_errno, "CreateEvent (\" ngx_cache_manager_event\ ") failed");
    Goto failed; }
    ...

At the beginning, there are still some events that are created, and you can see that there are some notifications that the worker process is restarted or shut down, and one is used to notify the event
Modify the status, and activate the event immediately. Then get the handle of the cache_manage mutex and create the Ngx_cache_manager_event
Event, this event is the command cache management thread exits, which is discussed in the function body later. After that, the worker process started with 3 main threads, namely
Worker threads, cache management threads, cache load Threads:

    ...
    if (Ngx_create_thread (&wtid, Ngx_worker_thread, NULL, log)! = 0) {
        goto failed;
    }

    if (Ngx_create_thread (&cmtid, Ngx_cache_manager_thread, NULL, log)! = 0) {
        goto failed;
    }

    if (Ngx_create_thread (&cltid, Ngx_cache_loader_thread, NULL, log)! = 0) {
        goto failed;
    }
    ...

The main thread of the worker process after startup enters an event processing loop:

    ... for (;;)

        {ev = WaitForMultipleObjects (3, events, 0, INFINITE);
        err = Ngx_errno;

        Ngx_time_update ();

        NGX_LOG_DEBUG1 (ngx_log_debug_core, log, 0, "worker waitformultipleobjects:%ul", Ev);
            if (ev = = wait_object_0) {ngx_terminate = 1;

            Ngx_log_error (ngx_log_notice, log, 0, "exiting"); if (ResetEvent (events[0]) = = 0) {ngx_log_error (ngx_log_alert, log, 0, "Rese
            Tevent (\ "%s\") failed ", WTEVN);
        } break;
            } if (ev = = wait_object_0 + 1) {ngx_quit = 1;
            Ngx_log_error (ngx_log_notice, log, 0, "gracefully shutting down");
        Break
            } if (ev = = Wait_object_0 + 2) {ngx_reopen = 1;

            Ngx_log_error (ngx_log_notice, log, 0, "reopening logs"); if (ResetEvent (events[2]) = = 0) {ngx_log_error (Ngx_log_alert,Log, 0, "resetevent (\"%s\ ") failed", WROEVN);
        } continue; } if (ev = = wait_failed) {ngx_log_error (Ngx_log_alert, log, err, "Waitform

            Ultipleobjects () failed ");
        Goto failed; }
    }
    ...

This event loop will handle the following 3 events, if re-open will set the restart location (may be processed later) take off the message and continue the loop if it is
terminate or exit will jump out of the loop, the flag will be set to jump out of the loop, if the call fails to enter the failure processing:

    .../* Wait threads */if (SetEvent (ngx_cache_manager_event) = = 0) {ngx_log_error (Ngx_log_alert, log
    , Ngx_errno, "SetEvent (\" ngx_cache_manager_event\ ") failed");
    } events[1] = Wtid;

    EVENTS[2] = Cmtid;

    Nev = 3; for (;;)

        {ev = WAITFORMULTIPLEOBJECTS (Nev, events, 0, INFINITE);
        err = Ngx_errno;

        Ngx_time_update ();

        NGX_LOG_DEBUG1 (ngx_log_debug_core, log, 0, "Worker exit WaitForMultipleObjects:%ul", Ev);
        if (ev = = WAIT_OBJECT_0) {break;
            } if (ev = = wait_object_0 + 1) {if (Nev = = 2) {break;
            } Events[1] = events[2];
            Nev = 2;
        Continue
            } if (ev = = Wait_object_0 + 2) {Nev = 2;
        Continue } if (ev = = wait_failed) {ngx_log_error (Ngx_log_alert, log, err, "WaitForMultipleobjects () failed ");