Analyze diagnostic database connection pooling issues

top-h-P PID can view CPU load, CPU waiting or blocking status

jmap-histo 2224 >20150411.txt, finally locate which method is causing the memory leak

Slowly the CPU load will drop down and the thread will break.

Yum Install-y Dstat

dstat-c: Show CPU Status

dstat-m: Show Memory Condition

dstat-d: Show Load Conditions

dstat-l: Show Load Conditions

dstat-n: Show Network conditions

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" height= "381" width= "553" alt= "Spacer.gif"/>

xxxxx The website has been spinning

The first case is: Is there a bottleneck in the load machine itself?

The second case is: Is there a bottleneck in the network?

The third scenario is: to the application server, verify that the application server has CPU queuing issues

If the pressure is too high, the CPU load is very high, that is, load average, there will be a large number of queues in the CPU this block can not get time slices, will also queue

The fourth case is: suspect to memory

Physical memory is not related, because it is a Java project

Suspect to JVM memory there is a frequent full GC generated, and when it pauses the entire application thread, look at the GC,JVM default configuration such as:

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ "Images/localimage.png") no-repeat center;border:1px solid #ddd; "height=" 347 "width=" "alt=" and "Spacer.gif"/>

The old age default is 4M, the maximum persistent default is 64M

Take a look at the Jstat-gcutil 2633 3000 5 command to see a large number of full GC, persistent generation has been over 99%

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" height= "" width= "530" alt= "" Spacer.gif "/>

The old age and the maximum persistent generation parameters in the JVM parameters of the catalina.sh in the bin directory under the tomcat1 of Tomcat (this virtual machine is a) are changed, Full GC will occur in the old age and lasting generations, specifically how much, refer to the following configuration, remove the comment #

#JAVA_OPTS = "$JAVA _opts-xms800m-xmx800m-xmn400m-xss1024k-xx:permsize=128m-xx:maxpermsize=128"

Save and restart Tomcat, and then use the Jmap-heap 2633 command to see the old and persistent generations of used, all under 20%, and then use jstat-gcutil 2633 5 command to see the full GC 2 times, Brush the Web page is not spinning

The fifth case is to see if there is a queue in the middleware thread pool, if there is a queue, you need to change the configuration of the thread pool, where to test, and then verify, until you determine whether it is a thread pool problem

The sixth case is: to the database connection pool here, see if there is a queue, how to see?

Web page circles, not timeout is queued, in the LoadRunner scene error log see 120s time out, should not be sure is the timeout caused, the second may be queued, then the line is only three blocks

One is to queue up here on the CPU:

CPU The time slice is a few one out of 10,000 of that kind of millisecond speed switch, can rule out the CPU this queuing problem, you can also view the CPU through the command

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" height= "107" width= "553" alt= "Spacer.gif"/>

In the CPU (s) See the value of us and Sy is a bit high, these two high can only indicate that the CPU processing performance is not high or the load is very spaced queue, load average the last 15 minutes of the payload is also very normal, WA is 0, there is no queue, can also be seen by the following command CPU no wait phenomenon

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" height= "148" width= "518" alt= "Spacer.gif"/>

The second is to queue in the middleware thread pool:

Configure the monitoring of the middleware thread pool, see if there is no queue, if not, remove the situation (this project proves that there is no problem here)

The last scenario is to queue up in the database connection pool:

In the database client input show processlist, the number of its own IP connection to the past database connection number, found to be 30 (in the process of the pressure to remove the red box three is not their own IP, the rest is the number of their IP connections, 33-3=30)

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/ Zh-cn/images/localimage.png ") no-repeat center;border:1px solid #ddd;" height= "427" width= "554" alt= "Spacer.gif"/> found in jdbc.properties no configuration database zodiac, in

/usr/local/tomcat1/webapps/xxxx/web-inf/applicationcontext.xml configuration, find Maxpoolsize

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" height= "203" width= "554" alt= "Spacer.gif"/>

Change 30 to 60, restart Tomcat, and then press again to see if the maximum number of connections becomes 60, and the maximum number of connections to the database becomes 60, indicating that the database connection pool caused by the program is not released, such as:

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" height= "428" width= "554" alt= "Spacer.gif"/>

The Application link library is normal:

A. establishing a database connection

B. executes the SQL statement and returns the result

c. To close a database connection

If the application does not close the database connection, the final step is not executed, so that the database connection will accumulate more and more, until the maximum number of connections reached, and then unable to establish a new connection, the page is stuck, loading does not come out

From the above see increased connection pool, the database show Processlist, added 60 after the problem again, this time we can be sure that the database of the link pool is not released after use caused.

This article is from the "surveillance Commands for SAR" blog, so be sure to keep this source http://rainbow9912.blog.51cto.com/9070528/1641679

Analyze diagnostic database connection pooling issues