The connection depletion solution is caused by unreasonable locks in the dbcp connection pool, and dbcp is unreasonable.

The connection depletion solution is caused by unreasonable locks in the dbcp connection pool, and dbcp is unreasonable.

Solution to connection depletion caused by unreasonable locks in the dbcp connection pool

The application reports an error, which indicates that the connection pool is full.

org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:241) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]    at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82) [dubbo-2.5.3.jar:2.5.3]    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_33]    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_33]    at java.lang.Thread.run(Thread.java:662) [na:1.6.0_33]Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:103) ~[commons-dbcp.jar:1.2.1]    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540) ~[commons-dbcp.jar:1.2.1]    at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74) ~[crmpub-jdbc-1.0.jar:1.0]    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]    ... 32 common frames omittedCaused by: java.util.NoSuchElementException: Timeout waiting for idle object    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174) ~[commons-pool-1.6.jar:1.6]    at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) ~[commons-dbcp.jar:1.2.1]    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) ~[commons-dbcp.jar:1.2.1]    ... 35 common frames omitted

The application stack blocks dozens of threads to the PoolableConnectionFactory of the dbcp pool. makeObject method. This method is waiting for the 0x0000000709a638a0 object lock, but the 0x0000000709a638a0 object has been executed by one of the threads for PoolableConnectionFactory. the makeObject is locked and the thread is BLOCKED after obtaining the lock. As a result, many other threads are BLOCKED.

"DubboServerHandler-132.121.95.136:28101-thread-300" daemon prio=10 tid=0x00007f02b4136800 nid=0x4954 waiting for monitor entry [0x00007f0286018000]  java.lang.Thread.State: BLOCKED (on object monitor)  at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)  - waiting to lock <0x0000000709a638a0> (a org.apache.commons.dbcp.PoolableConnectionFactory)  at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)  at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)  at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)  at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)  at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74)  at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)  at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)  at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417)  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)  at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)  at com.eshore.crm.service.common.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74)  at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)  at java.lang.reflect.Method.invoke(Method.java:597)  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)  at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)  at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)  at com.eshore.crm.service.sysmgr.sca.impl.SysMgrServiceImpl$$EnhancerByCGLIB$$bfd71326.qryBulletinList(<generated>)  at com.alibaba.dubbo.common.bytecode.Wrapper101.invokeMethod(Wrapper101.java)  at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)  at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)  at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)  at com.eshore.crmpub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90)  at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)  at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)  at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)  at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  at java.lang.Thread.run(Thread.java:662)  Locked ownable synchronizers:  - <0x000000070a120f00> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"DubboServerHandler-132.121.95.136:28101-thread-290" daemon prio=10 tid=0x00007f03203da800 nid=0x4948 runnable [0x00007f0286a22000]  java.lang.Thread.State: RUNNABLE  at java.net.SocketInputStream.socketRead0(Native Method)  at java.net.SocketInputStream.read(SocketInputStream.java:129)  at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)  at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)  at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)  - locked <0x0000000715a9ce70> (a com.mysql.jdbc.util.ReadAheadInputStream)  at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3140)  at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:597)  at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1085)  at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2494)  at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2527)  at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2309)  - locked <0x00000007159d1898> (a com.mysql.jdbc.JDBC4Connection)  at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)  at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)  at sun.reflect.GeneratedConstructorAccessor78.newInstance(Unknown Source)  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)  at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)  at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:419)  at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)  at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)  at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)  - locked <0x0000000709a638a0> (a org.apache.commons.dbcp.PoolableConnectionFactory)  at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)  at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)  at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)  at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)  at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74)  at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)  at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)  at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417)  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)  at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)  at com.eshore.crm.service.common.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74)  at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)  at java.lang.reflect.Method.invoke(Method.java:597)  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)  at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)  at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)  at com.eshore.crm.service.cachemgr.sca.common.DictCacheServiceImpl$$EnhancerByCGLIB$$e27e8e25.getDict(<generated>)  at com.alibaba.dubbo.common.bytecode.Wrapper93.invokeMethod(Wrapper93.java)  at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)  at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)  at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)  at com.eshore.crmpub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90)  at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)  at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)  at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)  at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  at java.lang.Thread.run(Thread.java:662)  Locked ownable synchronizers:  - <0x000000070a0f9218> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

We can see that the thread used to obtain the lock is blocked here, And the mysql handshake package is always waiting for reading, blocking the network IO.

 java.lang.Thread.State: RUNNABLE  at java.net.SocketInputStream.socketRead0(Native Method)  at java.net.SocketInputStream.read(SocketInputStream.java:129)  at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)  at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)  at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)  - locked <0x0000000715a9ce70> (a com.mysql.jdbc.util.ReadAheadInputStream)  at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3140)  at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:597)  at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1085)  at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2494)  at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2527)  at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2309)

This is because dbcp1.x locks the entire connection factory class object when creating a new connection object. Once a blocking occurs during the creation process, the entire pool will die. This is a big problem.

Solution: Upgrade dbcp2.x or tomcat jdbc or druid connection pool. We recommend that you use the druid connection pool, which has powerful built-in monitoring functions.

Thank you for reading this article. I hope it will help you. Thank you for your support for this site!