Record a Java Memory Leak analysis and a java leak

Record a Java Memory Leak analysis and a java leak
Current Environment Code address

Git address: https://github.com/jasonGeng88/java-network-programming

Background

Not long ago, a new project was launched. This project is a stress testing system, which can be viewed as a replaying Vocabulary (http request data) and continuously sending requests to the service, to achieve the purpose of the Pressure Test Service. During the test, everything went smoothly. After several bugs were fixed, they went online. When it was used by the first business party after the launch, it found a serious problem. After the application ran for more than 10 minutes, it received a large number of Full GC alarms.

To solve this problem, we first confirmed the content of the pressure test scenario with the business side. The number of word lists for playback is about 0.1 million, and the playback speed is about QPS for a single machine, according to our previous estimates, this is far below the limits that a single machine can afford. In principle, there will be no memory problems.

Online troubleshooting

First, we need to troubleshoot on the server. Use the jmap tool that comes with JDK to check the specific objects in the JAVA application, as well as the number of instances and the occupied size. The command is as follows:

Jmap-histo: live 'pid of Java' # To facilitate observation, write the output to the file jmap-histo: live 'pid of Java'>/tmp/jmap00

After observation, we found that some objects were instantiated by more than 0.2 million. According to the business logic, the most instantiated word table, that is, more than 0.1 million. How can there be more than 0.2 million, we did not find any display declaration instantiation in the code. So far, we need to further analyze the dump memory offline. The dump command is as follows:

jmap -dump:format=b,file=heap.dump `pid of java`

Offline Analysis

After downloading the heap. dump of dump from the server, we need to use tools for in-depth analysis. Recommended tools include mat and visualVM.

I personally prefer to use visualVM for analysis. In addition to offline dump files, it can also be integrated with IDEA to start applications through IDEA, analyze the CPU, memory, and GC of the application in real time (The visual GC plug-in must be installed in visualVM.). The tool is shown as follows (The data is not true only to demonstrate the effect.):

Of course, mat is also a very useful tool that can help us quickly locate memory leaks and facilitate our troubleshooting. As shown below:

Scenario Reproduction

After analysis, we finally locate the memory leakage problem caused by httpasyncclient. Httpasyncclient is an HTTP Toolkit provided by Apache. It mainly provides the reactor io non-blocking model and implements the asynchronous sending of http requests.

The following uses a Demo to briefly explain the cause of Memory leakage.

Httpasyncclient usage:

• Maven dependency

<dependency>    <groupId>org.apache.httpcomponents</groupId>    <artifactId>httpasyncclient</artifactId>    <version>4.1.3</version></dependency>

• HttpAsyncClient Client

public class HttpAsyncClient {    private CloseableHttpAsyncClient httpclient;    public HttpAsyncClient() {        httpclient = HttpAsyncClients.createDefault();        httpclient.start();    }    public void execute(HttpUriRequest request, FutureCallback<HttpResponse> callback){        httpclient.execute(request, callback);    }    public void close() throws IOException {        httpclient.close();    }}

Main logic:

The main logic of the Demo is as follows: first, create a cache list to save the request data to be sent. Then, the requests to be sent are retrieved from the cache list cyclically and sent to the httpasyncclient client.

The Code is as follows:

Public class ReplayApplication {public static void main (String [] args) throws InterruptedException {// create a playback client with Memory leakage ReplayWithProblem replay1 = new ReplayWithProblem (); // load 10 thousand pieces of request data into the cache List <HttpUriRequest> cache1 = replay1.loadMockRequest (10000); // start loop playback replay1.start (cache1 );}}

Playback client implementation (Memory leakage ):

Here, we use playback of Baidu as an example to create 10000 mock data records and put them in the cache list. During playback, a request is sent every 100 ms in a while loop. The Code is as follows:

/**
* Java learning and communication QQ group: 589809992 let's learn Java together!
*/

public class ReplayWithProblem {    public List<HttpUriRequest> loadMockRequest(int n){            List<HttpUriRequest> cache = new ArrayList<HttpUriRequest>(n);        for (int i = 0; i < n; i++) {            HttpGet request = new HttpGet("http://www.baidu.com?a="+i);            cache.add(request);        }        return cache;            }    public void start(List<HttpUriRequest> cache) throws InterruptedException {        HttpAsyncClient httpClient = new HttpAsyncClient();        int i = 0;        while (true){            final HttpUriRequest request = cache.get(i%cache.size());            httpClient.execute(request, new FutureCallback<HttpResponse>() {                public void completed(final HttpResponse response) {                    System.out.println(request.getRequestLine() + "->" + response.getStatusLine());                }                public void failed(final Exception ex) {                    System.out.println(request.getRequestLine() + "->" + ex);                }                public void cancelled() {                    System.out.println(request.getRequestLine() + " cancelled");                }            });            i++;            Thread.sleep(100);        }    }}

Memory analysis:

Start the ReplayApplication (After visual VM Launcher is installed in IDEA, you can directly start visual VM), Observe through visualVM.

• Startup status:

• Percentage of memory objects in the last three minutes in visualVM:

Note: $0 represents the object itself, and $1 represents the first internal class of the object. Therefore, ReplayWithProblem $1 indicates the callback class of FutureCallback in the ReplayWithProblem class.

From this, we can find that the FutureCallback class will be constantly created. Every time an http request is sent asynchronously, a callback class is created to receive the result. Logically, the result is normal. Don't worry. Let's continue.

• GC condition of three minutes before and after visualVM:

We can see that the old memory is constantly increasing, which is incorrect. In the memory, only the http Request body of the cache list should be maintained. As the number of requests continues to grow, it indicates that objects are constantly entering the old area. Combined with the memory objects above, it indicates that the FutureCallback object is not recycled in time.

However, after the http callback is complete, the reference relationship of the callback Anonymous class is gone, and the next GC should be recycled. After tracking the source code of the httpasyncclient request, we found that the internal implementation of httpasyncclient is to insert the callback class into the http request class, while the request class is placed in the cache queue, therefore, the reference relationship of the callback class is not removed, and a large number of callback classes are promoted to the old area, resulting in Full GC.

• Core code analysis:

Code optimization

Find the cause of the problem. Now we can optimize the code and verify our conclusion. BecauseList<HttpUriRequest> cache1The callback object is saved, so we cannot cache the request class. We can only Cache basic data and dynamically generate it when using it to ensure timely collection of the callback object.

The Code is as follows:

public class ReplayApplication {    public static void main(String[] args) throws InterruptedException {        ReplayWithoutProblem replay2 = new ReplayWithoutProblem();        List<String> cache2 = replay2.loadMockRequest(10000);        replay2.start(cache2);    }}

 

/**
* Java learning and communication QQ group: 589809992 let's learn Java together!
*/

public class ReplayWithoutProblem {    public List<String> loadMockRequest(int n){        List<String> cache = new ArrayList<String>(n);        for (int i = 0; i < n; i++) {            cache.add("http://www.baidu.com?a="+i);        }        return cache;    }    public void start(List<String> cache) throws InterruptedException {        HttpAsyncClient httpClient = new HttpAsyncClient();        int i = 0;        while (true){            String url = cache.get(i%cache.size());            final HttpGet request = new HttpGet(url);            httpClient.execute(request, new FutureCallback<HttpResponse>() {                public void completed(final HttpResponse response) {                    System.out.println(request.getRequestLine() + "->" + response.getStatusLine());                }                public void failed(final Exception ex) {                    System.out.println(request.getRequestLine() + "->" + ex);                }                public void cancelled() {                    System.out.println(request.getRequestLine() + " cancelled");                }            });            i++;            Thread.sleep(100);        }    }}

Result Verification

• Startup status:

• Percentage of memory objects in the last three minutes in visualVM:

• GC condition of three minutes before and after visualVM:

Which proves that our conclusion is correct. The callback class will be recycled in time in the Eden area. There is no sustained growth in the old area. This memory leakage problem is solved.

Summary

The first troubleshooting of memory leaks is often a bit confusing. We need to have the correct methods and means, coupled with easy-to-use tools to solve the problem. Of course, the basic knowledge of JAVA memory is also essential. At this time, you are the key to locating the problem. Otherwise, even if the tool tells you this error, you cannot locate the cause.

Finally, there is no problem with the use of httpasyncclient. We only need to understand its use scenarios, which often cause many problems and are caused by improper use. Therefore, the degree of understanding of the tool often determines the probability of a bug.