Performance case studies for projects through Jstack and Jmap

First, the problem found

With the CPU usage of the line machine, we can see that starting from April 8, with the increase of CPU usage in time, the end-use rate reaches 100%, which causes the online service to be unavailable, then recovers after restarting the machine.

Second, troubleshooting ideas

A simple analysis of the possible problems in the area, divided into 5 directions:

1. System itself code problems

2. Avalanche effect due to problems in the internal downstream system

3. Upstream system call Volume burst

4.http Ask a third-party question

5. Problems with the machine itself

Iii. start of troubleshooting

1. Review the log, no error logs found in the set, preliminary troubleshooting Code logic processing errors.

2. The internal downstream system was first contacted to observe their monitoring and found to be normal together. can rule out the impact of downstream system failures on us.

3. Look at the number of calls to the provider interface, compared to 7 days without a burst, excluding the issue of the number of business party calls.

4. View TCP monitoring, TCP status OK, can be excluded is the HTTP request third-party timeouts caused by the problem.

5. Check machine monitoring, 6 machine CPUs are on the rise, the same as each machine situation. troubleshoot problems with machine failures.

that is, the above method does not directly locate the problem.

Iv. Solutions

1. Restarted the 6 Taichung problem more serious 5 machines, first restore business. Keep a scene to analyze the problem.

2. View the current Tomcat thread PID

3. Check the system usage of the PID thread. TOP-HP 384

4. Found PID 4430 4431 4432 4433 threads occupy about 40% of the CPU

5. Convert these PID to 16, respectively 114E 114f 1150 1151

6. Download the current Java thread stacks sudo-u tomcat jstack-l 384>/1.txt

7. Query 5 for the corresponding thread condition, found to be caused by GC thread

8.dump Java Heap Data

Sudo-u Tomcat Jmap-dump:live,format=b,file=/dump201612271310.dat 384

9. Using the mat to load the heap file, you can see that the Javax.crypto.JceSecurity object occupies 95% of the memory space and initially locates the problem.

mat:http://www.eclipse.org/mat/

10. Look at the reference tree of the class and see that the Bouncycastleprovider object is holding too much. That is, the object is handled in the wrong way in our code, and the problem is fixed.

Five, Code Analysis

There's a piece in our code that says that.

This is the function of encryption and decryption, each run plus decryption will be a new Bouncycastleprovider object, put down the Cipher.getinstance () method.

Look at the implementation of Cipher.getinstance (), which is the underlying code implementation of the JDK, traced to the Jcesecurity class

Verifyingproviders each put will be remove,verificationresults only put, not remove.

See Verificationresults is a static map, which belongs to the Jcesecurity class.

So every run to add and decrypt will put an object to this map, and this map belongs to the class dimension, so it will not be recycled by GC. This results in a large number of new objects not being recycled.

Vi. Code Improvements

Set the problematic object to static, each of which holds one and does not create multiple times.

Vii. Summary of this paper

Do not panic in the face of online problems, first confirm the idea of troubleshooting problems:

1. View Logs
2. View CPU Status
3. Viewing TCP conditions
4. View Java threads, Jstack
5. View Java heap, Jmap
6. Find objects that cannot be recycled through the mat analysis heap file

Performance case studies for projects through Jstack and Jmap