Why ngix reload does not take effect after system restart

Why ngix reload does not take effect after system restart
Why ngix reload does not take effect after system restart

This is a rare problem that has plagued me for a long time. Although this problem is very simple, it still takes a lot of time to find the root cause. Now I will share the analysis process as follows.

Prerequisites: You need to understand the Linux Startup Process, Nginx process startup process, and process tracking.




I. Nginx reload Process Analysis:
After checking the documents on the official website and analyzing the Nginx source code, the reload process is roughly as follows.


1. Check whether the configuration is correct.
Equivalent to nginx-t
2. Open the log file.
Equivalent to nginx-s reopen
Because there are many log files, you need to open multiple files
3. Listen to the socket again.
Equivalent to nginx
This step initializes many things and focuses on the hash table.
4. Disable the old worker process.
Equivalent to nginx-s quit




2. nginx Process Analysis
1. First, understand the two processes of nginx
Master process, which is enabled by the root user, receives signals, and manages worker Processes
Worker Process, opened by nginx users, working process, responsible for handling http requests


2. starce traces the master process number and executes nginx-s reload during the process.
Main process tracking, because the reload process is the HUP signal sent by the system to the nginx MAIN PROCESS
# Starce-p 2298
......
Open ("/data/wwwlogs/access. xxx. log", O_WRONLY | O_CREAT | O_APPEND, 0644) =-1 EMFILE (Too upload open files)
Write (808, "09:50:22 [emerg] 2298"..., 124) = 124
......


3. Search for the System Limit file of the process as prompted.


Master process restrictions
# Cat/proc/2398/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited seconds
Max file size unlimited bytes
Max data size unlimited bytes
Max stacks size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited bytes
Max processes 127015 127015 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited bytes
Max file locks unlimited locks
Max pending signals 127015 127015 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited us


Worker Process restrictions
# Cat/proc/2300/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited seconds
Max file size unlimited bytes
Max data size unlimited bytes
Max stacks size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited bytes
Max processes 127015 127015 processes
Max open files 409600 409600 files
Max locked memory 65536 65536 bytes
Max address space unlimited bytes
Max file locks unlimited locks
Max pending signals 127015 127015 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited us




Supplemental Error Log:
10:48:05 [notice] 47386 #0: signal process started
10:48:05 [emerg] 2298 #0: open () "/data/wwwlogs/access_xxx.xxx.xxx.log" failed (24: Too workflow open files)




3. Solutions
1. Modify limits
Generally, optimization is performed in the following three aspects:
1. nginx. conf parameter planning and setting
Worker_rlimit_nofile: Limit the maximum number of files opened by a worker process:
No problems with online Configuration
Worker_rlimit_nofile 409600;


Second: system-level checks and settings
Is the configuration and modification of/etc/security/limits. conf. Refer to the Linux system resource limit summary.
No problems with online Configuration
* Soft nofile 655350
* Hard nofile 655350


Third: kernel-level check and settings:
Fs. file-max value size settings:
Large online Configuration
Fs. file-max = 6553600
Note: The default value of file-max is approximately 10% of the system memory (the system memory is calculated in kb)




2. verification takes effect
The above configurations are available in the early stages, but the restrictions on the master process are not modified after the server is restarted, however, after logging on to the server, it takes effect no matter whether the terminal ulimit-n is used to view or shut down the main nginx process, and then restart nginx.
The problem may occur when the linux system is started. That is to say, when the nginx main process is started, the above restriction configuration does not take effect. Later, I checked the information and found that the limits will be made only when login is executed after the system is started. the conf configuration takes effect, so you need to adjust the order.
The system startup process is as follows:
1. Read the/etc/inittab to read the default level. Assume that the default read level is 3.
2. Execute the initialization system script/etc/rc. d/rc. sysinit to initialize the script.
3. Run the/etc/rc. d/rc Script.
4. Run the/etc/rc. d/rc. local script, which is the last script started during the startup process.
The/bin/login logon user is executed. At this point, the/etc/profile will be executed only when login is started ,~ /. Bash_profile and ~ /. Bashrc, etc. The value found by ulimit-n at this time is not the value when the nginx process is started.
By default, the limits. conf configuration file will take effect when the user logs in. This is later than the nginx process is started. To make the configuration take effect before this, you need to add the following Configuration:
Cat/etc/rc. local
Ulimit-HSn 655350 (note that it is executed before nginx is started)
/Usr/local/nginx/sbin/nginx






Iv. Supplemental Optimization


The main difference is that the related parameters are increased.
1. kernel Optimization
Net. ipv4.tcp _ max_tw_buckets has a larger modification to reduce the kernel burden. iptable itself has an impact on Kernel performance.
# Ss-an | awk '{print $1}' | sort | uniq-c | sort-rn
15415 ESTAB
12979 TIME-WAIT
1961 FIN-WAIT-2
501 FIN-WAIT-1
234 LAST-ACK
32 SYN-RECV
11 LISTEN
3 CLOSING
1 SYN-SENT
1 State
1 CLOSE-WAIT


Online Configuration modification is as follows:
Net. ipv4.tcp _ max_tw_buckets = 18000


2. nginx Optimization
It is mainly a hash table. Other configurations have been optimized. There are several hash tables:
Server_names_hash can be added
Map_hash can be added
Types_hash is sufficient
The request header is not considered.
Variables_hash is sufficient.


Online Configuration modification is as follows:
Server_names_hash_max_size 512000;
Server_names_hash_bucket_size 64; (default)
Map_hash_max_size 204800;
Map_hash_bucket_size 64; (default)