PHP connection to Memcached short connection causes high CPU usage

It is found that the web server is abnormal in the monitoring, and the cpu and load will suddenly increase, and then it will return to normal in 10 minutes. Ssh connection, several php-cgi processes use 100% cpu. At that time, I suspected that some programs were not written, so I informed the development team to check them. As you can see, as the number of visits increases, several web servers found abnormal during monitoring yesterday, CpU and load will suddenly increase, and then it will return to normal in 10 minutes. Ssh connection, there are several Php-The cpu usage of the cgi process is 100%. At that time, I suspected that some programs were not written, so I informed the development team to check them. We can see that:

 

With the increase in access traffic, this has happened to several web servers yesterday, and the cpu and load are not working. It is always very high. Like this:

This is not normal. Use top to check resource utilization. % us and % sy are still in normal state, within 10%, that is, % si is always high, 50% ~ 90%.

It doesn't feel like the code is not written, because php + meMcAchEd+ MysqlArchitectureIt also runs normally on other projects.

The top display si refers to the soft interrupt, which will use cpu resources.

When I saw that the Soft Interrupt was too high, I began to wonder if the NIC driver was not good, because some NICs used only one cpu soft interrupt before, and I checked it.

So what exactly is the use of so many soft interruptions? It is suspected that it is a network connection.Stat-Anpo |Grep: 11211 | awk-F "" '{print $6}' |Sort|Uniq-C | sort-rn.TimeThere are many connections in the _ out status. Normally there shouldn't be so many.

Check the php Code directly here and find the connection Code as follows:

$ Handle = new Memcache ();
$ Handle-> connect ($ config ['host'], $ config ['Port']);

Connect is used. This is a short connection.

Logic: Why is soft interruptions so high?

Because php uses a short connection to connect to memcache, this method is to connect to memcached each time. After the read and write operations are completed, the connection is closed and the next time you need to read and write memcache, the connection is re-connected. Because php reads and writes memcache frequently, php naturally connects to memcache a lot. Creating a connection may incur soft interruptions. So we can see in top that % si will always be very high, 50% ~ 90%.

After determining the cause, modify the code:

$ Handle = new Memcache ();
$ Handle-> pconnect ($ config ['host'], $ config ['Port']);

Use pconnect, that is, a persistent connection to connect to memcached, so that every php process can repeat this connection as long as a connection is established.

Look at the server again. In the top, we can see that % si is less than 1%, and the cpu and load are all down.