Cause analysis of Ngix reload not effective after system restart
Cause analysis of Ngix reload not effective after system restart
This is a relatively rare, troubled me for a long time, although the problem is very simple, but to find the root cause or a lot of time, now the analysis process to share the following.
Prerequisite: The Linux system startup process, Nginx process startup process and process tracking must be understood.
One, Nginx reload process analysis:
After viewing the official website documents and combining nginx source code analysis, the reload process is generally concluded as follows.
1. Check that the configuration is correct
Equivalent to Nginx-t
2. Open the log file
Equivalent to nginx-s reopen
Multiple files need to be opened because of more log files
3, re-listen sockets
Equivalent to Nginx
This step initializes a number of things, focusing on the hash table
4, close the old worker process
Equivalent to Nginx-s quit
Second, nginx process analysis
1, first understand nginx two kinds of processes
Master process, root user open, receive signal, manage worker process
Worker process, Nginx user open, worker process, responsible for handling HTTP requests
2,starce tracks the main process number, and during execution of Nginx-s reload, the card is found in the check log file this block
Main process trace because the reload process is the system sends the HUP signal to the Nginx main process
#starce-P 2298
......
Open ("/data/wwwlogs/access.xxx.xxx.xxx.log", o_wronly| O_creat| O_append, 0644) =-1 Emfile (Too many open files)
Write (808, "2016/02/17 09:50:22 [Emerg] 2298" ..., 124) = 124
......
3, follow the prompts to find the system limit file for the process
Master Process Limit
# cat/proc/2398/limits
Limit Soft limit Hard limit Units
Max CPU time Unlimited unlimited seconds
Max File size Unlimited Unlimited bytes
Max Data size Unlimited Unlimited bytes
Max stack size 10485760 Unlimited bytes
Max core File size 0 Unlimited bytes
Max resident set Unlimited Unlimited bytes
Max processes 127015 127015 processes
Max open files 1024x768 4096 files
Max locked Memory 65536 65536 bytes
Max address space Unlimited Unlimited bytes
Max file Locks Unlimited Unlimited locks
Max pending signals 127015 127015 signals
Max msgqueue size 819200 819200 bytes
Max Nice priority 0 0
Max Realtime Priority 0 0
Max Realtime Timeout Unlimited Unlimited US
Worker Process Throttling
# cat/proc/2300/limits
Limit Soft limit Hard limit Units
Max CPU time Unlimited unlimited seconds
Max File size Unlimited Unlimited bytes
Max Data size Unlimited Unlimited bytes
Max stack size 10485760 Unlimited bytes
Max core File size 0 Unlimited bytes
Max resident set Unlimited Unlimited bytes
Max processes 127015 127015 processes
Max open files 409600 409600 files
Max locked Memory 65536 65536 bytes
Max address space Unlimited Unlimited bytes
Max file Locks Unlimited Unlimited locks
Max pending signals 127015 127015 signals
Max msgqueue size 819200 819200 bytes
Max Nice priority 0 0
Max Realtime Priority 0 0
Max Realtime Timeout Unlimited Unlimited US
Supplemental error log:
2016/02/17 10:48:05 [notice] 47386#0:signal process started
2016/02/17 10:48:05 [Emerg] 2298#0:open () "/data/wwwlogs/access_xxx.xxx.xxx.log" failed (24:too many open files)
Three, the solution
1, modify the limit
Generally from the following 3 aspects of tuning:
First: nginx.conf parameter planning and Setup
Worker_rlimit_nofile: Limit the maximum number of files that a single worker process opens:
No problem with on-line configuration
Worker_rlimit_nofile 409600;
Second: System-level checks and settings
is/etc/security/limits.conf configuration and modification, please refer to the Linux system resource limit summary
No problem with on-line configuration
* Soft Nofile 655350
* Hard Nofile 655350
Third: kernel-level checks and settings:
Fs.file-max the size of the value setting:
Large on-line configuration
Fs.file-max = 6553600
Note: The default value for File-max is approximately 10% of system memory (system memory in kilobytes)
2, validation is in effect
It is found that the above configuration has been configured earlier, but restart the server to find the main process limit has not been modified, but the login server after the terminal ulimit-n view or shut down the nginx main process after the restart Nginx is in effect, thus reasoning out
The problem may be in the Linux system boot process, that is, when the Nginx main process starts, the above restrictions are not in effect, and then check the data found that the system started after the login to the limits.conf configuration to take effect, so need to adjust the order.
According to the actual situation, the system startup process is as follows:
1. Read/etc/inittab to read the default level hypothesis: Read to the default level is 3
2. Initialize the script/etc/rc.d/rc.sysinit to initialize the system script
3. Then execute the/ETC/RC.D/RC script
4. Execute the/etc/rc.d/rc.local script, which is the last script launched during startup.
The/bin/login logged-on user is executed at the end. At this point the system boot process is complete, login will be executed/etc/profile,~/.bash_profile and ~/.BASHRC, and so on, at this time the value of ulimit-n found is not the value of the Nginx process startup.
The default user login will make the limits.conf configuration file take effect, this is more than the Nginx process started later, to make the configuration effective before this, need to supplement the configuration as follows:
Cat/etc/rc.local
ULIMIT-HSN 655350 (note the execution before nginx boot)
/usr/local/nginx/sbin/nginx
IV, supplemental optimization
Mainly related parameters were adjusted a little.
1, kernel optimization
Net.ipv4.tcp_max_tw_buckets Modify the larger, reduce the burden of the kernel, the iptable itself has an impact on the kernel performance
# Ss-an |awk ' {print $} ' |sort |uniq-c |sort-rn
15415 estab
12979 time-wait
1961 Fin-wait-2
501 Fin-wait-1
234 Last-ack
Syn-recv
LISTEN
3 CLOSING
1 syn-sent
1 state
1 close-wait
The online modification configuration is as follows:
Net.ipv4.tcp_max_tw_buckets = 18000
2,nginx optimization
Mainly hash table, other configuration has been optimized, hash table has the following several
Server_names_hash can add
Map_hash can add
Types_hash enough
Request header does not consider
Variables_hash enough
The online modification configuration is as follows:
Server_names_hash_max_size 512000;
Server_names_hash_bucket_size 64; Default
Map_hash_max_size 204800;
Map_hash_bucket_size 64; Default
http://www.bkjia.com/PHPjc/1103189.html www.bkjia.com true http://www.bkjia.com/PHPjc/1103189.html techarticle after the system restarts Ngix reload does not take effect causes analysis system restarts Ngix reload does not take effect cause analysis This is a relatively rare, troubled me for a long time, although the problem is very simple, but ...
