The shared memory is also the most basic inter-process communication mode provided in Linux, it creates a continuous linear address space in the memory through MMAP and shmget system calls, and the memory can be released through munmap or shmdt system calls. The advantage of using shared memory is that when multiple processes use the same shared memory, after any process modifies the content in the shared memory, other processes can obtain the modified content by accessing the shared memory.
Nginx defines the ngx_shm_t struct to describe a shared memory,
Typedef struct {// point to the actual address of the shared memory u_char * ADDR; // The length of the shared memory size_t size; // The name of the shared memory ngx_str_t name; // The ngx_log_t object ngx_lot_t * log that records logs; // indicates whether the shared memory has been allocated with a flag. If the value is 1, ngx_uint_t exists;} ngx_shm_t;
There are two methods to operate the ngx_shm_t struct: ngx_shm_alloc (implemented based on MMAP) is used to allocate new shared memory, while ngx_shm_free (implemented based on munmap) is used to release existing shared memory.
The main way for nginx processes to share data is to use the shared memory. Generally, it is created by the master process. After the master process fork has a sub-process, all processes start to use the data in this memory.
Nginx channel ngx_channel_t is a common tool for communication between the nginx master process and the worker process. It is implemented using a local socket. The socketpair method is used to create sockets used between parent and child processes.
Int socketpair (int d, int type, int protocol, int SV [2]);
Generally, before the Parent and Child processes communicate, socketpair is called to create a set of sockets. After the fork method is called to create a child process, the SV [1] socket is disabled in the parent process, the child process closes the SV [0] socket.
The ngx_channel_t channel structure is the message format for communication between the master process and the worker sub-process defined by nginx. As follows:
Typedef struct {// command ngx_uint_t command in the TCP Message passed; // process ID, which is generally the idngx_pid_t PID of the Process sending the command; // indicates the serial number ngx_int_t slot of the sending command between the ngx_processes process array; // The communication socket handle ngx_fd_t FD;} ngx_channel_t;
The message format is so simple that nginx only uses this channel to synchronize the status between the master process and the work process. This can be obtained quickly for commands already defined by the command members, as shown below:
// Open the channel. The command that must be sent before communication using the channel
# Define ngx_assist_open_channel 1
// Close the opened channel, that is, close the socket
# Define ngx_pai_close_channel 2
// The receiver is required to exit the process normally.
# Define ngx_1__quit 3
// The receiver is required to forcibly terminate the process
# Define ngx_cmd_terminate 4
// Ask the recipient to re-open the file that has been opened by the Process
# Define ngx_1__reopen 5
The master process sends commands through the socket generated by socketpair, that is, the socketpair method is called every time a process is to be derived. In the ngx_spawn_proces method of the nginx dispatch process, a TCP-based socket is first derived.
Nginx encapsulates four methods: ngx_write_channel, and ngx_close_channel.
The ngx_write_channel method used to send messages.
Ngx_int_t ngx_write_channel (ngx_socket_t S, ngx_channel_t * Ch, size_t size, ngx_log_t * log );
Here, the S parameter is the TCP socket to be used, the CH parameter is a message of the ngx_channel_t type, the size parameter is the size of the ngx_channel_t struct, and the log parameter is the log object.
Message reading method ngx_read_channel
Ngx_int_t ngx_read_channel (ngx_socket_t S, ngx_channel_t * Ch, size_t size, ngx_log_t * log );
The worker process uses the ngx_add_channel_event method to add the socket that receives the channel message to epoll, when the parent process message is received, the child process calls back the corresponding handler method through the epoll event to process the channel message.
Ngx_int_t ngx_add_channel_event (ngx_cycle_t * cycle, ngx_fd_t FD, ngx_int_t event, ngx_event_handler_pt handler );
The cycle parameter is the ngx_cycle_t core struct that every nginx process must possess. The FD parameter is the socket used to receive messages. The event parameter is the type of event to be detected. Here it must be epollin; the handler parameter points to the method used to read messages.
Void ngx_close_channel (ngx_fd_t * FD, ngx_lot_t * log );
The FD parameter is the socket array mentioned above.
Signal nginx defines an ngx_signal_t struct to describe the behavior of the received signal:
Typedef struct {// int signo signal to be processed; // The string name corresponding to the signal char * siname; // The nginx command char * name corresponding to this signal; // After receiving the signo signal, the handler method void (* Handler) (INT signo) will be called back;} ngx_signal_t;
An array signals is also defined to define all signals that the process will process, for example:
ngx_signal_t signals[] = {{ngx_signal_value(NGX_RECOFIGURE_SIGNAL),“SIG” ngx_value(NGX_RECONFIGURE_SIGNAL),“reload”,ngx_signal_handler},…}
After the signals array is defined, the ngx_init_signals method initializes all signals in the signals array. ngx_init_signals actually calls the callback method of the sigaction method to register signals.
Ngx_int_t aggregate (nx_log_t * log) {ngx_signal_t * SIG; struct signaction SA; // traverses the signals array and processes each ngx_signal_t struct for (Sig = signals; sig-> signo! = 0; sig ++) {ngx_memzero (& SA, sizeof (struct, sigaction); // set the signal processing method to handler method SA. sa_handler = sig-> handler; // set all values in SA to 0 sigemptyset (& SA. sa_mask); // callback Method for registering signals if (sigaction (sig-> signo, & SA, null) =-1) {ngx_log_error (ngx_log_emerg, log, ngx_errno, "sigaction (% s) failed", sig-> signame); Return ngx_error;} return ngx_ OK ;}
In this way, the process can process the signal. The signal is set and generated before the fork () function is called, so work money once can all be affected. Of course, in general, we do not send control information to child processes such as working processes, but mainly want to monitor the sending of the parent process. After the parent process receives the signal and processes it accordingly, depending on the situation, check whether the signal is sent to all other sub-processes.
Process Synchronization mainly uses atomic operations, semaphores and file locks. The spin lock can be implemented based on atomic operations. Based on atomic operations, semaphores, and file locks, nginx encapsulates a mutex lock at a higher level for convenience.
Only integer types of atomic variables can be used for atomic operations, including the unsigned integer ngx_atomic_uint_t and the signed integer ngx_atomic_t. Both types Use the volatile keyword to tell the C compiler not to perform optimization.
Nginx provides two methods to use atomic operations to modify and obtain integer variables:
Ngx_atomic_cmp_set and ngx_atomic_fetch_add. Both methods can be used to modify the value of the atomic variable, while the ngx_atomic_cmp_set method can also compare the value of the atomic variable.
Static ngx_inline ngx_atomic_uint ngx_atomic_cmp_set (ngx_atomic_t * Lock, ngx_atomic_uint_t OLC, ngx_atomic_uint_t set)
The ngx_atomic_cmp_set method compares the old parameter with the lock value of the atomic variable. If they are equal, the lock is set to the set parameter, and the method returns 1. If they are not equal, if no modification is made, 0 is returned.
Static ngx_inline ngx_atomic_int_t ngx_atomic_fetch_add (ngx_atomic_t * value, ngx_atomic_int_t add)
The ngx_atomic_fetch_add method adds the value of the atomic variable value to the parameter add and translates the value.
Spin lock
Based on atomic operations, nginx implements a spin lock. A spin lock is a non-sleep lock. That is to say, if a process attempts to obtain a spin lock and finds that the lock has been acquired by another process, it does not cause the current process to sleep, it is always in the executable state. Every time the kernel is scheduled to execute this process, it continuously checks whether the lock can be obtained. When the lock is not obtained, the code of this process will always be executed in the spin lock code, knowing that other processes have released the lock and the current process has obtained the lock, the code will continue to be executed.
It can be seen that the spin lock is mainly set for the multi-processor operating system. The shared resource protection scenario to be solved is that the process uses the lock for a very short time. Most of the nginx worker processes should not go to sleep, because they are very busy and there may be 100,000 or even millions of TCP connections on the epoll of this process to be processed, once a process is sleep, it must wait for another period of time to wake up. The load consumption caused by frequent process switching may not be acceptable to users.
The following describes how to implement the spin lock method ngx_spinlock Based on atomic operations.
It has three parameters. The lock parameter is the lock expressed by the Atomic variable. When the lock value is 0, the lock is released, when the lock value is not 0, the lock is held by a process. The value parameter indicates that the lock is not held by any process, setting the lock value to value indicates that the current process holds the lock. The third parameter spin indicates that the current process is in a multi-processor system. When the ngx_spinlock method does not get the lock, the current process is in one scheduling of the kernel, the time for this method to wait for other processors to release the lock. Let's take a look at its source code:
/** Copyright (c) Igor Sysoev * copyright (c) nginx, Inc. */# include <ngx_config.h> # include <ngx_core.h> // function: Implementation of the spin lock method ngx_spinlock Based on atomic operations // parameter explanation: // lock: lock expressed by atomic variables // value: Flag, whether the lock is occupied by a process // spin: In a multi-processor system, when the ngx_spinlock method does not get the lock, the time when the current process waits for other processors to release the lock in a kernel scheduling voidngx_spinlock (ngx_atomic_t * Lock, ngx_atomic_int_t value, ngx_uint_t spin) {# If (ngx_have_atomic_ops) // supports atomic operations ngx_uint_t I, n; // it remains in the loop until the lock (; ;) {// If lock is 0, no other process holds the lock, set the lock value to the value parameter, indicating that the current process holds the lock if (* Lock = 0 & ngx_atomic_cmp_set (lock, 0, value) {return ;} // if it is a multi-processor system if (ngx_ncpu> 1) {/* under a multi-processor, when the lock is found to be occupied by other processes, the current process does not immediately give up the CPU processor in use, but waits for a period of time to see if the processes on Other Processors release the lock, which reduces the number of inter-process switching times. */For (n = 1; n <spin; n <= 1) {// as the number of waits increases, the interval between actually checking locks is getting bigger and bigger for (I = 0; I <n; I ++) {/* ngx_cpu_pause is an instruction provided by many architecture systems for spin locks. It tells the CPU to be in the spin lock wait state. Generally, a CPU puts itself in the energy-saving state, reduce power consumption. However, the current process does not give up the processor in use. */Ngx_cpu_pause (); //}/* check whether the lock is released. If the lock value is 0 and the lock is released, set it to value, the current process successfully holds the lock and returns */If (* Lock = 0 & ngx_atomic_cmp_set (lock, 0, value) {return ;}}} /* 'the current process gives way to the processor but is still in the executable State, so that the processor gives priority to scheduling other executable processes. In this way, when the process is re-scheduled by the kernel, in the for loop code, you can expect other processes to release the lock. */Ngx_sched_yield () ;}# else # If (ngx_threads) # error ngx_spinlock () or ngx_atomic_cmp_set () are not defined! # Endif}
When releasing the lock, the nginx module must use the ngx_atomic_cmp_set method to set the atomic variable to 0.
Semaphores nginx only uses semaphores as simple mutex locks. using semaphores as mutex locks may cause process sleep. Do not elaborate.
A file lock is a file read/write mechanism that allows only one process to access one file at any specific time. Using this mechanism can make the process of reading and writing a single file more secure. Do not elaborate.
The mutex implemented by nginx is based on atomic operations, semaphores, and file locks. nginx encapsulates a mutex lock at a higher level, which is easy to use. Many nginx modules only accept it. The following describes how to operate the mutex lock in step 5:
Ngx_shmtx_create initialize mutex lock
Ngx_shmtx_destory
Ngx_shmtx_trylock tries to obtain the mutex lock unimpeded. 1 indicates that the mutex lock is obtained successfully, and 0 indicates that the mutex lock fails to be obtained.
Ngx_shmtx_lock gets the mutex lock by blocking the process. The mutex lock is held when the method returns.
Ngx_shmtx_unlock release mutex lock
When obtaining the mutex lock, you can either use the ngx_shmtx_trylock method that does not block the process, or use the ngx_shmtx_lock method to tell nginx that the code can continue to be executed only after the mutex lock is held. They all implement the mutex structure by operating the ngx_shmtx_t structure. Let's take a look at the members of ngx_shmtx_t.
Typedef struct {# If (IF) // atomic variable lock ngx_atomic_t * lock; # If (ngx_have_posix_sem) // when semaphore is 1, it indicates that the semaphore ngx_uint_t semaphonre that may be used to obtain the lock; // SEM is the semaphores lock sem_t SEM; # endif; # else // when the file lock is used, FD indicates the file handle ngx_fd_t FD; // name indicates the file name u_char * Name; # endif/* Number of spin times, indicating the time to wait for other processors to release in the spin state. Implemented by file locks, spin has no significance */ngx_uint_t spin;} ngx_shmtx_t;
The ngx_shmtx_t structure involves two macros: ngx_have_atomic_ops and ngx_hve_poix_sem. These two macros correspond to three different implementations of mutex.
1st implementations: When atomic operations are not supported, the file lock is used to implement the ngx_hmtx_t mutex lock. At this time, it only has FD and name members. The two members use the File locks described above to provide blocking and non-blocking mutex locks.
2nd implementations, supporting atomic operations but not semaphores.
3rd implementations. The operating system also supports semaphores while supporting atomic operations.
The only difference between the two solutions is the effect of the ngx_shmtx_lock method during execution. That is to say, the support for semaphores only affects the way the ngx_shmtx_lock method of the blocked process holds the lock. When semaphores are not supported, the ngx_shmtx_lock lock is consistent with the spin lock described above. After semaphores are supported, ngx_shmtx_lock will wait for other processors to release the lock within the time specified by spin, if the lock has not been obtained before reaching the spin upper limit, sem_wait will be used to make the current process sleep. The process will be awakened only after other processes return to the lock kernel. Of course, after the ngx_shmtx_lock method runs for a period of time, if other processes never give up the lock, the current process may forcibly obtain the lock, this is also because nginx should not use the sleep lock of the blocking process.
