Remember that the Tomcat process is too CPU intensive to troubleshoot records

This article primarily records the Tomcat process, which causes the CPU to take up too high a number of problems with TCP connections. Problem Description

Under Linux, a Tomcat Web service has a very high CPU footprint and the top display results over 200%. The request cannot be answered. Repeated restart is still the same phenomenon. Problem Troubleshooting 1. Get process information

The JPS command provided through the JDK can quickly detect the JVM process,

JPS PID 2, view jstack information

Jstack PID

Found a large number of log4j thread blocks, in waiting lock state

Org.apache.log4j.Category.callAppenders (org.apache.log4j.spi.LoggingEvent) @bci =12, line=201 (Compiled frame)

Search for relevant information and found a deadlock problem with the log4j 1.x version.

Finding the problem, adjust the log4j configuration, open the error level log only, and restart Tomcat. The block thread in the stack disappears, but the process CPU usage is still high. 3. Further investigation

To analyze the CPU footprint of each thread, you need to introduce a great God contribution script that calculates the CPU usage of each thread in the Java process.

#!/bin/bash

typeset top=${1:-10}
typeset pid=${2:-$ (Pgrep-u $USER java)}
typeset tmp_file=/tmp/java_${ Pid}_$$.trace

$JAVA _home/bin/jstack $pid > $tmp _file
ps h-eo user,pid,ppid,tid,time,%cpu--sort=%cpu-- No-headers\
        | tail-$top \
        | awk-v "pid= $pid" ' $2==pid{print $ "\ T" $} "
        | while read line;
Do
        typeset nid=$ (echo "$line" |awk ' {printf ("0x%x", $)} ')
        typeset cpu=$ (echo "$line" |awk ' {print $} ')
        Awk-v "cpu= $cpu" '/nid= ' "$nid" '/,/^$/{print $ \ t ' (isf++ "": "cpu=" CPU "%");} ' $tmp _file done

rm-f $tmp _ File

Script Application scope

Because the%CPU data in PS comes from/proc/stat, the data is not real-time, but depends on how often the OS updates it, typically 1S. So the data that you see is inconsistent with the information that is jstack. That's why ~ but this information is very useful for troubleshooting problems that are caused by a few threads on a continuous load, because these few threads will continue to consume CPU resources, even if there is a time lag, Anyway, it's all caused by these threads.

In addition to this script, the easiest way to do this is to look at the resource usage of each thread in the process with the following command after the process ID is detected

Top-h-P PID

Gets the PID (thread ID) from here, converts to 16, and then looks up the thread information for the object in the stack information.

By using the above method, the corresponding thread CPU occupancy rate of the Tomcat process is found to be 80%, much less than the top given 200%+

The description does not exist for a long time CPU-intensive thread, it should belong to a lot of transient CPU dense calculation. Further suspicion is that the JVM is not out of memory and frequent GC causes.

JSTAT-GC PID

Found no exceptions to JVM memory usage, and a significant spike in GC times

After checking the memory, because it is a network program, further troubleshoot network connections. 4, the problem positioning

Query the TCP link for the tomcat corresponding port, found a large number of eastablish links, and some other state of the connection, total 400 +.

NETSTAT-ANP | grep Port

Further looking at the source of these connections, it is found that there is a large number of background threads on the application side of the Tomcat service that are frequently polled, causing the service to be full of tomcat connections and unable to continue receiving requests. 5. Root cause Analysis

The direct trigger is the client polling, requesting an exception, continuing the rotation, and the client continues to have a new background thread joining the polling queue, which eventually results in a full server-side Tomcat connection.

However, the reason for the client request exception is that the root cause is the thread block caused by the log4j of the server.