Because the number of connections to redis files is insufficient, the listen sock is always readable and the CPU is full.

A few days ago, I encountered an online redis instance with a CPU Full and basically couldn't process normal requests. At first I thought it was a problem elsewhere, later, grep "Max open files"/proc/'pidof redis-Server'/-R was used to start redis. Ulimit-N is only 1024, so it cannot accept new connections.

A large number of sudden requests during peak hours resulted in redis connections exceeding 1024, so that listen sock continued to read and accept failed, resulting in CPU Full, resulting in a serious avalanche. For example, in simple reproduction, ulimit-N 20 modifies the number of opened files in the current session, starts a server program, and sends a TCP connection that exceeds the limit to it, at this time, the listening socket will certainly return a readable handle every time you select/epoll. In this way, you can call the accept continuously, and then the following error occurs immediately, namely, emfile:

Accept failed. errno: 24, errmsg: Too too open files.

However, in the implementation of accept, the handling method with insufficient handles is: Stay in the next processing, rather than disconnect the TCP connection, which also makes sense, because the next session may be closed.

However, this will cause the listening socket to constantly have readable messages, but the accept cannot be accepted, so that the listen backlog is full; thus, the subsequent connections are rst.

Here, let's talk a little bit more about memcached's handling of this situation. If memcache returns emfile upon accept, it will immediately call listen (SFD, 0 ), that is, set the backlog of the listener socket waiting for the accept queue to 0, so as to reject this part of the request, reduce the system load, and protect itself. Pretty good.

This article from the "Technical Achievement dream" blog, please be sure to keep this source http://hxl2009.blog.51cto.com/779549/1543616

Because the number of connections to redis files is insufficient, the listen sock is always readable and the CPU is full.