Python error hint: [Errno] Too many open files analysis and resolution

Background

Recently found an error in the work, the following error occurs frequently when executing a multithreaded scan script

Httpconnectionpool (host= ' t.tips ', port=80): Max retries exceeded with URL:/index.php (caused by newconnectionerror (' <requests.packages.urllib3.connection.httpconnection object at 0x7f29d4081150>: Failed to establish a new Connection: [Errno] Too many open files ',))

More bizarre ...

Analysis

Bo master himself in some small projects often need to carry out multi-threaded batch scanning to collect data, such as the ability to collect data, of course, the more threads the better, so I usually use 3,000 threads to scan, in determining the bandwidth is sufficient, the system hardware configuration is high enough to continue the problem, This reminds me of the limitations of the system.

Through the search engine I found a description of the system limitations, execution

$ ulimit-n1024

The result is 1024, and the system is limited to opening 1024 files at the same time, which is obviously too little.

Solve

The solution to this problem is simple, modify the following file directly

sudo vim/etc/security/limits.conf

The last two lines of code to add to this file

* Soft nofile 10240* hard Nofile 10240

Of course, this number according to their own needs to make changes can be saved after logging off and re-login just fine.

Python error hint: [Errno] Too many open files analysis and resolution