Precautions for using a supervisor

Precautions for using a supervisor

It has always been the use of supervisor to manage service processes on linux. Recently, some colleagues said that a service seems to be faulty. Let's check it out. After going up, I found that the response to a service was really slow, so I restarted it with the supervisor. However, the problem was discovered during the restart, and the operation can be completed in one or two seconds. Now it takes several minutes! This is very unscientific. Check the reason!
At first, I wondered whether there was a problem with the file system and whether I/O would be too high. After exclusion, check the supervisor process and find that the cpu is high. This is strange. The cpu usage of the supervisor on other servers is very low. Why? Then compare the processes managed by the supervisor on the two machines. Finally, we found that on the faulty machine, some processes managed by the supervisor had a large number of frequent stdout outputs (tornado is used here, default logs are output to stdout ). When many processes output a large number of stdout at the same time, the supervisor will be exhausted in processing the output data and recording logs, and eventually lead to slow processing by the supervisor.

After the logger output mode of the managed program is adjusted, the supervisor will return to normal immediately.
Therefore, when using a supervisor, it is best to avoid a large number of frequent stdout or stderr outputs in the managed program. Otherwise, problems may occur in the management.