Too many open files error

This is a creation in Article, where the information may have evolved or changed.

As we all know, recently I have been imitating Binux's pyspider shy group to run for some time online. Later, I added something new, such as a proxy pool. Look at the code more and more reliable, the result suddenly one day, found that the crawl stopped, and then to look at the log:

2015/08/12 23:18:22 Post http://api.duoshuo.com/posts/import.json: dial tcp: lookup api.duoshuo.com: too many open files

As a rookie, I don't know what this is. Later with Google to search, found that this is a full of Linux sockets. In the directory /proc//fd/ , there are all open file identifiers related files for the process, and sockets are also part of the file. Under Default Linux, the maximum number of socket concurrency per process is 1024, which is limited to open files. I started by entering the directory and using commands to ll | wc -l view the number of socket connections for the current process.

This look is that there is something to apply for not released, decisively restarted the service. But I still don't know where the release is. Can only use Google to continue to check, I know less, check a lot of things, the conclusion to say a bit.

In the /proc/net/tcp file, the current computer is saved the TCP protocol connection, my side of the server results are as follows:

 sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode276: BE848368:80E5 2F80CFB7:0050 08 00000000:000019E4 00:00000000 00000000  1000        0 3748204 1 ffff8800182c1c00 116 4 30 10 -1

Then is the conversion IP, which local and remote IP is 16 binary, and is inverted row, the conversion of the local IP is 104.131.132.190:32997 (be848368:80e5), the remote IP is 183.207.128.47:80 ( 2f80cfb7:0050), stating that my computer is connected to this IP is not properly disconnected. Check this IP, he is I use the shrine agent inside the one provided, the agent that part of the new code to write a problem.

Later learned that the netstat command was also, and better than this. I generally also use -p parameters, this can see the connection related processes, and other parameters I do not understand, we would like to know about Google. netstat -p | grep haixiuzuthis will be able to see the connection of my shy group program. One of the results shown at that time is the following table. In addition, it is also found that after the end of a crawl there will be 30 close_wait status of the connection, where 118.144.80.201 the IP has 20, the amount is not small, he is more say the IP of the API, to save to say when the HTTP also did not release the connection.

Proto Recv-Q Send-Q Local Address         Foreign Address     State      PID/Program nametcp        1      0 104.131.132.190:60059 118.144.80.201:http CLOSE_WAIT 14397/haixiuzu

On-line check, the great God said if Recv-Q and Send-Q have not 0 of the wrong, this is the sending and receiving queue, stating that has been waiting, generally need two hours operating system will automatically release this resource, but according to my observation, two hours will not be normal, or will not increase to 1024.

The problem is generally fixed, then the location and repair. Again to see the TCP connection 3 times handshake and disconnect 4 times handshake, presumably is the handshake shook half did not continue to hold. and went to see Golang net/http bag, disconnect is resp.Body.Close() , I did not write this, holding a try to change the attitude, found that the TCP connection is back to normal ...

I built this wheel can still, let me learn some of the TCP fur, or a sense of accomplishment.

Please refer to the complete source code in this article.

Original link: Too many open files error, reproduced please specify the source!