Record one php high-load troubleshooting experience

Record the php high load troubleshooting experience. in the last project, deamon was run in the background to handle the task. There has been no problem before. after a monitoring report such as adding task processing time, deamon's machine experienced high load, with an average of more than 20 and a backlog of tasks.

First, use strace to view system calls:

Strace-tt-T-c-p 31420

Strace-c found that the system time is mainly spent on rt_sigprocmask. This function is the system's signal shielding function, which is generally used in the signal processing program to shield the signal, ensure that the signal processing program is not interrupted. Rt_sigprocmask is a linux system call function, which is encapsulated by glibc instead of being called externally.

Further details about starce call:

The rt_singprocmask call has a feature that restores the blocked word immediately after the blocked word is set. This may be called by the php signal processing function.

Search for sigprocmask in the php source code and find that the place where singprocmask is called in the php source code is mainly in the pnctl extension, and the deamon program uses the pnctl extension to create and manage sub-processes. The signal processing function of php is pcntl_signal_dispatch.

Then, we added logs to verify our ideas and confirmed that the function was frequently called. This leads to the analysis of the php signal processing mechanism.

The specific analysis process is omitted because the analysis result is as follows:

1. Php tick mechanism:

Phpmanul's explanation of tick is:

A tick is an event that occurs for every N low-level tickable statements executed by the parser within the declare block. the value for N is specified using ticks = N within the declare block's ctictive section.

Not all statements are tickable. Typically, condition expressions and argument expressions are not tickable.

The event (s) that occur on each tick are specified using the register_tick_function (). See the example below for more details. Note that more than one event can occur for each tick.

Generally, tick and register_tick_function work together to trigger the register_tick_function registration event every time n underlying commands with tick = n are executed.

2. php signal processing:

Php manages processes and signals through pcntl extension. pcntl uses the tick mechanism as the signal callback mechanism. you can use declare to allow callback in php.

Through the source code analysis of pcntl, the signal processing mechanism of pcntl is as follows:

1), signal registration

2), signal processing:

From the analysis, we can see that in php, the tick value declared by declare is called regularly. when the tick value is low (ticks = 1 defined in this program ), A large number of signal processing programs will be triggered. calling rt_sigprocmask in this program will result in a lot of cpu waste and high machine load.

So why does php adopt this signal processing mechanism?

When I checked the php manual, I found this introduction:

PCNTL now uses ticks as the signal handle callback mechanic, which is much faster than the previous mechanic. this change follows the same semantics as using "user ticks ". you use the declare () statement to specify the locations in your program where callbacks are allowed to occur. this allows you to minimize the overhead of handling asynchronous events. in the past, compiling PHP with pcntl enabled wocould always incur this overhead, whether or not your script actually used pcntl.

Important: You use the declare () statement to specify the locations in your program where callbacks are allowed to occur. this means that you can specify the code region where the signal is to be captured, and do not process the signal outside the region.

The solution to the problem can be summarized as follows:

1) use declare to specify the region where the application captures signals,

2) for programs that do not require signal processing, do not declare declear (ticks = n), and thus do not trigger the call of the signal processing program; you can manually call the signal processing program pcntl_signal_dispatch to replace php's timed detection.

After verification, any of the above methods can solve the problem. The server load is reduced to normal.

Summary:

1) php uses a mechanism similar to the Linux kernel interrupt processing mechanism (hard interrupt + soft terminal) to process signals. pay attention to the frequency of detection signals to avoid wasting cpu performance.

2) because it is not recommended to use pcntl extension to manage the creation process in cgi, this mechanism will not affect the cgi of webserver.