Mysql com. mysql. jdbc. StatementImpl $ CancelTask Memory leakage problem and solution, mysqljdbc

Mysql com. mysql. jdbc. StatementImpl $ CancelTask Memory leakage problem and solution, mysqljdbc

Recently, I have been responsible for the maintenance and construction of the company's SMS gateway. As the company's business develops, the reliance on SMS is becoming more and more serious. The number of SMS messages sent every day has increased by more than 40 million to million. Because Java is used for sending the underlying layer, memory problems are inevitable when the pressure increases. Memory leakage occurs when the sending volume is close.

After checking the system running, we found that:

1) After restarting the system for 3-4 hours, it is found that the system is unstable;

2) after 3-4 hours, the out of memory error occurs: java. lang. OutOfMemoryError: GC overhead limit exceeded.

After this problem is found, the monitoring log is obtained directly through JMS, and the system memory collection is abnormal. The gc pressure is very high and there is a significant memory backlog.

Then, analyze the system memory down and find that there is indeed a memory backlog:

This is a scheduled task of mysql, which is mainly used for query timeout. For example, when the system executes an SQL query, jdbc will give you a time-out period. To ensure the time-out, you can disable statement to enable a scheduled task to protect the time-out. If the SQL statement does not respond to execution timeout, the cancel task will close the task. The default timeout value of c3p0 is 25 s (<setting name = "defaultStatementTimeout" value = "25000"/>), which means that when a large number of SQL statements are executed within 25 s, to a certain extent, the cancel task will cause system instability. (This is not the root cause, but the representation)

However, the system itself uses mysql as the sending queue, which has a lot of mysql operations. If it is only a small amount of effort to optimize the code.

In a short period of time, stable business is the most important task. Forced to use a temporary solution based on the above situation.

Temporary plan

Based on the above judgment, it can be basically determined that the cancel task execution in a group of threads should form a peak value, supporting the JVM heap. However, at present, too much debugging cannot be performed on the service to be stopped. Therefore, the JVM memory is scaled more than doubled when the system is disconnected, hoping that the system can cross a wave of memory peaks. As a result, when the JVM memory is adjusted to 2 times of the fault, the system memory is restored to normal operation, but the maximum cancel task will occupy more than 2 GB of memory. Although it will also be recycled, it is not a good way to keep expanding the memory.

Solution

The mysql jdbc 5.1.6 version used by the system immediately decompile the mysql code and find the following problems. Because the timer of the cancel task is statically stored in the connection, it means that if statement normally queries the structure, it is the root cause of failure that the business cannot recycle the memory of the cancel task.

After identifying the problem with mysql jdbc, the fundamental solution should be to check whether mysql jdbc has solved this bug.

Fix this bug in version 5.1.11. After the bug is updated, the memory leakage fault is solved. Http://dev.mysql.com/doc/relnotes/connector-j/en/news-5-1-11.html

Thanks to Guo Bo for his discussion and assistance with ruikang, so that the fault can be solved successfully!