Remember that a Web application CPU is too high

LZ develops an in-house application for querying hive data usage. There is always a high CPU situation after deployment, and local testing is difficult to reproduce. The previous several times have been repeated through the direct restart after the use of, because it is internal use, reboot is not a big impact (of course, each reboot is incidentally changed the bug, add some monitoring, or modify some parameters).

Today, the case of high CPU consumption (the machine is 16 cores), after running for a few days, took up the situation of 200-300cpu, and then quickly increased, occupied to 700-1000. Randomly check the status of the application state.

First, we look at the GC and see if we are doing FGC.

(at the beginning of the CPU when the height of the FGC is 0, 9 times to locate the problem to intercept the figure, the previous no reservation)

The discovery of applications is not FGC, but frequent ygc.

YGC also has anomalies, S1 and S0 regions are jumping from 0 to 99%

Look at the heap size can be found in the memory of the young area is constantly from 0 to 99, and the old area is slowly increasing, has not reached the state of FGC. But the follow-up is expected to continue to rise, resulting in FGC channel, the application can not provide services.

After discovering YGC frequent about 3 hours, finally began the frequent Fgc,old district full.

The following are the case of heap size and FGC:

You can see that almost 10 seconds old area is full to cause a FGC, and the old area size is 10G (how scary, 10 g 10 seconds to run out)

From the memory and GC angles only see YGC frequently resulting in the FGC channel (10 seconds at a time). However, such a situation can be used to provide services, but only the higher CPU (if FGC in some applications should be inaccessible).

Check the state of the memory is not the old area is too small, is used full of the cause of the constant FGC, because it is obviously big enough, if 10G is not enough, then the application must be problematic.

After that, we checked which threads took up a very high CPU (this should be done after discovering that there are no frequent FGC, not enough experience)

To view thread CPU usage for a process:

Get a few threads that are CPU intensive and turn into 16 binary

Using Jstack to see thread stack information within a process (Jstack 24820 > Abc.log)

Vim edit Abc.log to find the 16 binary number of threads obtained above

The discovery is a thread that uses the Com.trilead.ssh2.StreamGobbler class. This is an SSH third-party package.

Subsequent checks on other CPU-intensive threads, in addition to GC occupancy, most of them are ssh occupied.

The scenario is that each time a query using HIVEQL a thread, SSH to the remote server, execute the query content, return the results.

If more than one user makes multiple queries at the same time (one user can make multiple queries), multiple threads of SSH will occur.

Two scenarios for subsequent processing:

Scenario One, replace the existing third-party SSH tool to see if there is any better

Scenario Two, modify the code. Consider SSH to maintain the session (check the time limit of SSH session), unified account, all users in the background using a unified account to log on to the remote server to execute the query. Only one session is used, and a re-connection is found after the session expires. Instead of every query to build a login once to get session, run out after the shutdown, I believe can reduce the problem caused by SSH.

(PS: Because there are still other projects on hand, so still restart the application first a burst, after the completion of the Hive permissions control, the background unified user after the implementation of optimized SSH processing)

OOM, CPU consumption issues, JVM tools use these are from the novice programmer constantly improve to experience, and hope that they become better.

Description of common tools with JVM: http://my.oschina.net/feichexia/blog/196575