Remember once through Memory analyzer analysis of memory leak resolution process

Status Description:
Recently the new version of the project, not long before the project will be dead. Condition is in a state of suspended animation. Index query is very slow, almost no operation, slowly stuck to death.
Then, when we make the release, we can only replace or add the class file based on the war package that we've made before.

Situation comparison and Analysis:
Because the code before a large reorganization, the submission of more code, so by rolling back version of the way to solve, more difficult. One is because the results of rectification can not be obliterated in vain; the other is that many files, by manual comparison to find, more difficult.

Solution One:

Before, I have been questioning the way the current project is packaged, so the first thing I suspect when this happens is Jenkins and the production of Tomcat servers. When I connect to production through a fortress machine, I find that launching an application through Jenkins will start two two tomcat processes.

Then, this seems to strengthen my view, immediately found the operation of the Victoria, but confirmed that there is no problem. One is started with the root user, and one is started with a tomcat user. A daemon process, an application process.


Solution Two:
Troubleshoot problems with the server and start thinking about the program head-on.
Re-issue the item with the problematic version, dump down the log, and then quickly roll back the observation. The dump log for a single machine has 5 G:

Through Memory analyzer analysis, the following analysis results are shown in the leak supects view:

As shown above, there are three categories of questions A, B, C, and some others, the type D.


Let's take a look at the first question (it turns out that the first few questions are the same problem)

Start with the details to see:

The figure above shows a clearly problematic thread: The address is 0x7c8ff3df0 and the name is pool-16-thread-1.
As you can see from the accumulated Objects in Dominator tree view, there is a large list object in the thread that has a large number of MySQL JDBC objects stored in the list object.

We want to look at the data in the JDBC object inside each other. Next we open the view of the open dominator for entire heap:

Locate the thread named 0x7c8ff3df0 pool-16-thread-1. As you can see, this thread takes up a lot of space and is not released. A layer of open objects inside:

The data here is the data for one of our user tables. So it can be concluded that within a thread, a list holds a large number of user objects retrieved from the database.
Think of here, we went to see the B, C problem description, is the same problem. It is estimated that at different points in time, the part has been recycled through the GC.

Then, I just looked at the details of question a, and next we looked at the StackTrace stack of a.

As shown above, the problem is obvious, in 112 lines of service, in the call of the FindByCustomerID method, there is scanning the entire table operation. After analysis, find the corresponding location, the corresponding code is:

CustomerID = Stringutils.isnotblank (CustomerID)? Customer.getcustomer (). Getcustomerid (): null;
                        Customer Oldcutsomer = Customerservice.findbycustomerid (CustomerID);

Obviously, when the process comes here, the CustomerID is always empty, then the Customerservice.findbycustomerid (CustomerID) method performs a scanning full table operation. Because of the large amount of data in the table, the development of an indexed query that the user is supposed to execute every time is actually a slow query and needs to return full table data. A large number of threads come over, consuming a large number of database connections, resulting in insufficient number of database connections, and each thread processing requires a lot of time, causing the project to be in a state of suspended animation.

Summary and Analysis:

1, code management is not standardized, resulting in this farce;

2, the development of the right to be too small, resulting in a tiny operation requires a variety of cross-sectoral applications;

3, take into account each other's face, some problems can not be a profound reflection, there is no better plan to prevent.