Actual combat PHP Workerman CPU too high caused by the business delay troubleshooting and resolution

Business Scenarios

Terminal through the Internet TCP protocol through the Nginx reverse proxy server and the Workerman process in the online PHP server communication, is a long connection, the demand for real-time high.

System and application Environment

# uname -r3.10.0-693.11.1.el7.x86_64# cat /etc/centos-releaseCentOS Linux release 7.4Workerman   version:3.5.5PHP         version:5.6.36# php -m[PHP Modules]eventphpiredis这里只列出了高并发相关的已经加载的模块

Phenomenon

The terminal is opened by means of a sweep code, and it is found that there is a delay of about 5-10 seconds after the sweep code to start the next process. The experience has become very bad.

Troubleshooting Procedures

Because the terminal is communicating with Workerman, it is a direct view of the application

Workerman Real-time connection status

With the htop instruction, it was found that the CPU core (CPU 1) occupied by Workerman was particularly high.

Supposedly, the number of CPU cores that have just been increased should improve the CPU high problem ah. However, careful observation, the machine's business is divided into traditional PHP classes and Workerman, according to the official manual, Workerman not with PHP-FPM have too much influence. In fact, it is also reflected that the terminal connected with Workerman will be delayed, the same time, the relevant PHP access is not affected, unless the entire server CPU is super high.

Started to focus on the disk IO and network IO bottlenecks, but when I pulled out the relevant performance monitoring, I found that was not the reason. Although there is a log-writing behavior, there is no pressure on the SSD disk.

Immediately seeking the help of Workman official technology group, through the status page and related monitoring system instructions, raised the question:
What is the CPU-intensive operation of the business? How much is the CPU so high?

Check the CPU status of each core through Mpstat, and find that CPU cores running Workerman do exist high CPU.

Then, immediately use the Strace command to track the situation

strace命令是一个集诊断、调试、统计与一体的工具，我们可以使用strace对应用的系统调用和信号传递的跟踪结果来对应用进行分析，以达到解决问题或者是了解应用工作过程的目的。strace -ttp 进程号

Each of these lines is a system call from which it is easy to see what the process is doing, where it can navigate to the process card, whether the card is connected or read the network data, etc.

When sending a message with the terminal is a go back, there is a fd=10 operation quite frequent and refreshing very fast.

Use Lsof to track this process

# lsof -p 24373 | grep 10

As you can see, this is the native communication to the Redis host (no default 6379 port number is used)

Is it the redis bottleneck that caused it?

Re-confirm that the Redis configuration is connected as follows

Less than 50,000 PPS per second in the intranet, far from reaching the limit of PPS

It turns out that this is not because of the Redis service bottleneck.

Positioning problems

Communicate with the developer to know that this is the operation of the terminal equipment query. and access to the same data, so crazy access to redis, resulting in high CPU utilization, it is easy to delay.

At this point, the basic positioning problems, the rest will be left to the development of optimization.

Solution Solutions

Subsequent re-update solution: Please wait ...

Actual combat PHP Workerman CPU too high caused by the business delay troubleshooting and resolution