Tracking of high tomcat cpu usage and system load

2011-09-06

For an online 8-core linux server with a normal load of 8, the current CPU load is too high, the maximum load is over 30, and the average load is around 20, which has lasted for nearly a week, the specific service that consumes CPU resources is tomcat_ SC, which occupies up to 720% CPU resources.
Use jconsole to trace

Change the startup settings of catalina. sh:

 

$ CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8933 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=$server_ip";

 

import java.rmi.registry.LocateRegistry;
import javax.management.MBeanServer;
import javax.management.remote.JMXConnectorServer;
import javax.management.remote.JMXConnectorServerFactory;
import javax.management.remote.JMXServiceURL;
import java.io.IOException;
import java.lang.management.ManagementFactory;
import java.net.MalformedURLException;

public class JmxTest {
public static void main(String[] args) {
MBeanServer mbeanServer =
ManagementFactory.getPlatformMBeanServer();

JMXServiceURL url = null;
try {
url = new JMXServiceURL(
"service:jmx:rmi://localhost:12199/jndi/rmi://localhost:8933/jmxrmi");
} catch (MalformedURLException e) {
e.printStackTrace();
}

JMXConnectorServer connectorServer =
null;
try {
connectorServer = JMXConnectorServerFactory.newJMXConnectorServer(url, null, mbeanServer);
} catch (IOException e) {
e.printStackTrace();
}

try {
System.setProperty("java.rmi.server.randomIDs", "true");
LocateRegistry.getRegistry(8933);
connectorServer.start();
} catch (IOException e) {
e.printStackTrace(); 
}

}
}

Hostname-I is 127.0.0.1

After the test server is configured, connect to the local machine using jconsole. Enter the password of the test server account to connect to the server. It is also feasible to change the host name without changing the host name.

The server port is enabled online and the local port is enabled. telnet can be connected, but the jconsole cannot be connected. Check google and say:

The jvm you're trying to connect to actually exposes * two * ports, the one specified via-Dcom. sun. management. jmxremote. port, and some other one. the 2nd one is random, but jconsole wants to connect to it, so if you have a firewall, and you 've only opened up the above port, you're hosed.

Is it possible to open only one port? Must all ports on the internet server be open to the Intranet? Continue tracking.

Jmap jconsole jstack is a java-based jmx problem tracking tool. You can learn how to locate program problems such as memory overflow program deadlocks.

Use jmap to view memory Conditions

Jmap-histo: live pid

A small number of self-built services

The jstack trace stack does not show any reason.

Continue to jconsole to investigate google's "jconsole remote set random port to certain" and find a decent article:

Http://www.componative.com/content/controller/developer/insights/jconsole3/

So I wrote servlet to register the specified port.

Try to write the java file on the test machine:

Java. rmi. AccessException: Cannot modify this registry error

Comment out the startup settings of catalina. sh-Dcom. sun. management. jmxremote. port = 8933.

Finally, I gave up jconsole.
Use java. lang. management... ThreadMXBean
Use the jsp page provided by Xinyang to analyze performance problems. It mainly depends on thread blocking.
Main Code:

ThreadMXBean tm = ManagementFactory.getThreadMXBean();
tm.setThreadContentionMonitoringEnabled(true);
<%
long [] tid = tm.getAllThreadIds();
ThreadInfo [] tia = tm.getThreadInfo(tid, Integer.MAX_VALUE);

long [][] threadArray = new long[tia.length][2];

for (int i = 0; i < tia.length; i++) {
long threadId = tia[i].getThreadId();

long cpuTime = tm.getThreadCpuTime(tia[i].getThreadId())/(1000*1000*1000);
threadArray[i][0] = threadId;
threadArray[i][1] = cpuTime;
}

The following thread problems are detected:
Thread ID: 89
Thread Name: http-6080-Processor73
Thread State: RUNNABLE
Thread Lock Name: null
Thread Lock Owner Name: null
Thread CPU Time: 35678 sec
Stack Info: (depth: 31)
+ Java. util. HashMap. get (HashMap. java: 303)
+ Com. netqin. javaske. server. nqrs. CloudSecurityCommand. writePkgsLog (CloudSecurityCommand. java: 466)
Mongocom.netqin.mongoke.server.nqrs.cloudsecuritycommand.exe cute (CloudSecurityCommand. java: 153)
+ Com. netqin. javaske. server. BaikeServer. service (BaikeServer. java: 64)
+ Sun. reflect. GeneratedMethodAccessor33.invoke (Unknown Source)
CPU usage reaches 35678 seconds. By about 50000 seconds in the afternoon, the CPU usage of tomcat reaches 200%.
The code analysis shows that hashmap is used as the class object in the singleton bean. During multi-threaded access, hashmap is not a thread-safe non-Singleton, causing problems. No problem found in a few months after code correction