Linux program resource restrictions

1 -- Problem Description

Program architecture Background:
N Business threads. Receives requests transmitted from network threads, processes them according to business logic, and finally returns the processing results to the network thread in bytes format, so that the network can return the processing results in the original path.
1 network thread. The Network thread is responsible for underlying network operations such as listen, accept, send, and Recv. This thread determines whether the FD has a complete request. If the request is complete, it is passed directly to the business thread for processing.

Symptom description:
1. The service has not received any request;
2. Refer to the/proc/FD directory of the network thread and run the "l-l | WC-L" command to find that the number of file descriptors has reached 20480 and remains unchanged;
3. Run the "strace-P network thread PID" command to find the following problems:

accept(8, 0x7ffff42743f0, [16])         = -1 EMFILE (Too many open files)accept(8, 0x7ffff42743f0, [16])         = -1 EMFILE (Too many open files)accept(8, 0x7ffff42743f0, [16])         = -1 EMFILE (Too many open files)accept(8, 0x7ffff42743f0, [16])         = -1 EMFILE (Too many open files)accept(8, 0x7ffff42743f0, [16])         = -1 EMFILE (Too many open files)accept(8, 0x7ffff42743f0, [16])         = -1 EMFILE (Too many open files)

Based on this, it can be determined that the link currently maintained by the server has exceeded the process resource limit.

2 -- view the resource limits of Running Processes

On a Linux server, the default file descriptors are 1024. You can view them by running the ulimit-N command.

For large-scale network servers, 1024 file descriptors mean that up to 1024 clients can be accepted, obviously not enough. The backend programmer needs to increase the value as appropriate. For details, refer to other articles in the blog.

When looking for a problem, we may want to know the maximum number of file descriptors allowed by the current process, or verify that the modification of the maximum file descriptor has taken effect. We adopt the following methods:

cat /proc/PID/limits | grep “Max open files”

The result is as follows:

motadou@96.108:/home/motadou> cat /proc/22706/limits | grep "Max open files"Limit                     Soft Limit           Hard Limit           Units     Max open files            204800               204800               files

It is learned that a process with a PID of 22706 can open up to 204800 files.

3 -- Analysis of "/proc/Pid/limits"

motadou@96.108:/home/motadou> cat /proc/22706/limits Limit                     Soft Limit           Hard Limit           Units     Max cpu time              unlimited            unlimited            seconds   Max file size             unlimited            unlimited            bytes     Max data size             unlimited            unlimited            bytes     Max stack size            8388608              unlimited            bytes     Max core file size        419430400            419430400            bytes     Max resident set          7011418112           unlimited            bytes     Max processes             62929                62929                processes Max open files            204800               204800               files     Max locked memory         65536                262144               bytes     Max address space         8322990080           unlimited            bytes     Max file locks            unlimited            unlimited            locks     Max pending signals       62929                62929                signals   Max msgqueue size         819200               819200               bytes     Max nice priority         0                    0                    Max realtime priority     0                    0                    Max realtime timeout      unlimited            unlimited            us