Tomcat-solution to ultra-high CPU utilization

A text message alert was sent at noon, indicating that the CPU usage of the Web server is high.

It is occupied by Java processes, and the internal system traffic is very small (less than 300). Therefore, high CPU utilization on the server is abnormal, and there are not many log errors. After killing and restarting, the CPU usage will soar, up to 500%

Like getting stuck in an endless loop, someone mentioned that a newly added piece of code in git is not a rigorous loop statement, so searching for that statement is a statement for deleting files, for example:

if (file.exist())     while (file.delete())          xxx          xxxfi

It should be, so why cannot it be deleted?

1. The file does not exist, but the code has passed the judgment,

2. Permission issues. If the account has no permissions, it will be in this endless loop.

Lenovo again-Tomcat started with root one week ago during debugging. Later, when it was automatically deployed, the script failed to kill the original process, but a new one was created, so two Tomcat servers appeared, one of them runs as root and calls the corresponding file, so even if the root process is killed, it also has a substantial impact-the permissions of the file directory occupied by the processes whose identities run have changed. (Changed to root), so other accounts cannot be deleted, which leads to an endless loop.

Solution:

1. Change the code

2. Change the original attribute of the relevant file directory.

Two pitfalls:

The endless loop of code is not rigorous enough.

Never start a process with a fixed user as the root user (due to misoperations, be cautious)

Other ideas:

1. Check logs to see many records of deletion failures. This should be noted to better find the cause.

2. jstat is used to analyze the JVM status. The interval between jstat-gcutil PID (vmid) and the number of milliseconds, for example:

[[email protected] ~]# jstat -gcutil 14503 1000 4  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT  43.75   0.00   0.00  76.49  85.93    148   17.511     1    0.618   18.12943.75   0.00   0.00  76.49  85.93    148   17.511     1    0.618   18.12943.75   0.00   0.00  76.49  85.93    148   17.511     1    0.618   18.12943.75   0.00   0.00  76.49  85.93    148   17.511     1    0.618   18.129

This reflects GC statistics. For details, refer to the use of jstat.

This article is from the "O & M Story" blog and will not be reproduced!

Tomcat-solution to ultra-high CPU utilization