A problem is found online recently. After the program runs on the server for three weeks, the new log file cannot be created. In addition, this program needs to use curl to access the back-end HTTP service call always fails. The log file cannot be created because it is not a key service and has not been paid much attention at the beginning. However, access to the backend HTTP service fails. Once the monitoring alarm rings, we first try to solve the problem and restart all the programs to solve the problem. Then, immediately check the cause of the problem.
Before the program is restarted, it is naturally suspected that there is a problem with the backend HTTP service (although the alarm for this service is not reported). A colleague tries to call curl through the command line on the server, no problem (this also resolves DNS and firewall issues, and does not need to be pinged or telnet ).
It is normal to query the CPU, memory, and disk space used by the server when a problem occurs.
I suddenly associate the problem that the new log file cannot be created with the problem that curl fails to call the backend HTTP service. Both of them need to create a file handle (curl needs to create a socket handle ), is it because the file handle is exhausted?
In the test environment, when a program is under high pressure and the command line uses "lsof-c program name | wc-l", it is indeed found that more and more open file handles exist. The problem is found. Next we will find that the socket handle in a function is not closed after it is created. After close (fd) is added, the number of opened file handles does not increase in the test environment again, and the problem is solved.
The maximum number of file handles that can be opened in the program is 65535. Therefore, this problem was not mentioned before and was frequently released by the system (twice a week, it does not appear at all. However, when the release frequency slows down, the problem arises.
Therefore, in Linux program development, in addition to the CPU usage, memory usage, and hard disk space, you also need to pay special attention to the release of file handles opened by the program.
PS: the maximum number of programs to update the opened file handle is as follows:
[Cpp]
# Include <sys/time. h>
# Include <sys/resource. h>
Void max_fd ()
{
Struct rlimit rlim, rlim_new;
If (getrlimit (RLIMIT_NOFILE, & rlim) = 0 ){
Printf ("RLIMIT_NOFILE Old Soft (% d), Hard (% d) \ n", rlim. rlim_cur, rlim. rlim_max );
If (rlim. rlim_max <65535 & rlim. rlim_max! = RLIM_INFINITY)
{
Rlim_new.rlim_max = 65535;
}
If (rlim. rlim_cur <65535 & rlim. rlim_max! = RLIM_INFINITY)
{
Rlim_new.rlim_cur = 65535;
}
If (setrlimit (RLIMIT_NOFILE, & rlim_new )! = 0 ){
Rlim_new.rlim_cur = rlim_new.rlim_max = rlim. rlim_max;
(Void) setrlimit (RLIMIT_NOFILE, & rlim_new );
Printf ("RLIMIT_NOFILE New Soft (% d), Hard (% d) \ n", rlim_new.rlim_cur, rlim_new.rlim_max );
}
Else www.2cto.com
{
Printf ("RLIMIT_NOFILE New Soft (% d), Hard (% d) \ n", rlim_new.rlim_cur, rlim_new.rlim_max );
}
}
}
Author: huyiyang2010