A troubleshooting case for a Java memory leak

This is a more typical Java memory usage problem, the positioning process is relatively straightforward, but the new person still a little reference value, so the record a bit.

Here's a look at how to step through the process of analyzing and locating a specific code without knowing the system code
(for new references and self-review)

A preliminary phenomenon

Business system consumption MQ message slow, backlog of 200多万条 messages, through Jstat observed business system FULLGC more frequent, to the end simply Oom:

?

Further analysis

Now that you know that there is a problem with memory usage, you need to know which objects are consuming a lot of memory.

A lot of people will think of the heap dump down again with tools such as mat, but the dump heap takes a long time, and the file is huge, and then dragged back from the server to the local import tool, this process is too much to do what is best not to do so.

With more lightweight online analysis, using Jmap to see the surviving objects (jmap-histo:live [PID]), you can see that the elements in Hashtable are more than 50 million, taking up approximately 1.5G of memory:

?

Navigate to Code

Now that you know the problem is hashtable, then you need to locate what code is causing the

The next step is to see what code goes into Hashtable's crazy put data, and then use artifact Btrace to track the stack of Hashtable.put calls.

First write Btrace script Tracinghashtable.java:

import com.sun.btrace.annotations.*;import static com.sun.btrace.BTraceUtils.*;@BTracepublic class TracingHashTable {        /*指明要查看的方法，类*/        @OnMethod(            clazz="java.util.Hashtable",            method="put",            [email protected](Kind.RETURN))        public static void traceExecute(@Self java.util.Hashtable object){                println("调用堆栈！！");                jstack();        }}

Then run:
BIN/BTRACE-CP Build 4947 Tracinghashtable.java

See a lot of similar call stacks

?

It can be seen that after receiving the message query the inbound code caused, business method call Ibatis to MySQL JDBC driver execution statement when put a lot of properties into Hashtable.

Through the above troubleshooting has been basically positioned by that block of code, the next is to open the Code project for White-box transformation, the corresponding code optimization (not within the scope of this article.) Some of the PID inconsistencies in the diagram are not tangled, some of the system restarted again.

For more content please visit the starting blog http://jenwang.me

Further communication:

-Email:[email protected]

-For some topics of this blog interested in, hope to further exchange, please add QQ Group: 2825967

-more technical exchange sharing in the Circle "Architecture Talk" , talk to old drivers about the internet's cutting-edge technologies, architectures, tools, solutions, etc.

A troubleshooting case for a Java memory leak