Troubleshooting of java thread Blocking

Troubleshooting of java thread Blocking

The worker I developed is blocked once every several months and has never been checked for any problems. Today, I finally got this knot. Summarize the solution process and share it with you.

First, run the jstack command to display all the thread stacks of the process. After obtaining the thread dump file, search for your worker name.

 

"DefaultQuartzScheduler_Worker-10" prio=10 tid=0x00007f55cd54d800 nid=0x3e2e waiting for monitor entry [0x00007f51ab8f7000]   java.lang.Thread.State: BLOCKED (on object monitor)at com.jd.chat.worker.service.impl.NewPopAccountSyncServiceImpl.addAccounts(NewPopAccountSyncServiceImpl.java:86)- waiting to lock <0x0000000782359268> (a com.jd.chat.worker.service.impl.NewPopAccountSyncServiceImpl)at com.jd.chat.worker.service.timer.AccountIncSyncTimer.run(AccountIncSyncTimer.java:114)at com.jd.chat.worker.service.timer.AbstractTimer.start(AbstractTimer.java:44)at com.jd.chat.worker.service.timer.AbstractTimer.doJob(AbstractTimer.java:49)at com.jd.chat.worker.web.context.StartAppListener$TimerJob.execute(StartAppListener.java:188)at org.quartz.core.JobRunShell.run(JobRunShell.java:202)at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)- locked <0x0000000783641c68> (a java.lang.Object)

Soon I found out which line of the thread was blocked. However, the real cause of the problem cannot be identified by the wine bottle information. Here we recommend a tool called tda. bat. My colleague gave it to me. There should be downloads online. Import the dump file to tda. Find the blocked thread. The blocked thread is red.

 

This software is good because after you find the blocked thread, a more detailed thread stack will be blocked at the bottom of the interface. Truncate part of the stack.

 

at org.mariadb.jdbc.MySQLPreparedStatement.execute(MySQLPreparedStatement.java:141)at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeUpdate(SqlExecutor.java:80)at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteUpdate(MappedStatement.java:216)at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeUpdate(MappedStatement.java:94)at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.update(SqlMapExecutorDelegate.java:457)at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.update(SqlMapSessionImpl.java:90)at org.springframework.orm.ibatis.SqlMapClientTemplate$9.doInSqlMapClient(SqlMapClientTemplate.java:380)at org.springframework.orm.ibatis.SqlMapClientTemplate$9.doInSqlMapClient(SqlMapClientTemplate.java:1)at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:200)at org.springframework.orm.ibatis.SqlMapClientTemplate.update(SqlMapClientTemplate.java:378)at com.jd.im.data.dataresource.ImSqlMapClientTemplate.retriedWithoutAnyInterventionUpdate(ImSqlMapClientTemplate.java:169)at com.jd.im.data.dataresource.ImSqlMapClientTemplate.update(ImSqlMapClientTemplate.java:137)at com.jd.chat.dao.impl.WriteDaoImpl.update(WriteDaoImpl.java:21)at com.jd.chat.zone.service.impl.GroupServiceImpl.updateRoute(GroupServiceImpl.java:766)at com.jd.chat.worker.service.impl.NewPopAccountSyncServiceImpl.addAccounts(NewPopAccountSyncServiceImpl.java:267)- locked <0x0000000782359268> (a com.jd.chat.worker.service.impl.NewPopAccountSyncServiceImpl)

This is a truly useful stack! It tells me that when the program executes the SQL statement, the SQL statement is deadlocked, and the thread is blocked. It also provides more useful information, that is, the SQL statement that causes the deadlock. The last row of the stack indicates the SQL statement that causes the deadlock.

But must I use this tool to find the specific reason? Of course the answer is NO!

Tell everyone how to find the real cause of blocking without using tools!

Just found the thread stack through the "BLOCKED" keyword and found its thread name "DefaultQuartzScheduler_Worker-10 ". OK, then, change the last 10 to 1, that is, "DefaultQuartzScheduler_Worker-1", and then search for the entire process stack with this keyword.

 

"DefaultQuartzScheduler_Worker-1" prio=10 tid=0x00007f55cd2aa000 nid=0x3e25 runnable [0x00007f51b02c0000]   java.lang.Thread.State: RUNNABLEat java.net.SocketInputStream.socketRead0(Native Method)at java.net.SocketInputStream.read(SocketInputStream.java:129)at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)at java.io.BufferedInputStream.read(BufferedInputStream.java:317)- locked <0x0000000791370d50> (a java.io.BufferedInputStream)at org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:82)at org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:92)at org.mariadb.jdbc.internal.common.packet.RawPacket.nextPacket(RawPacket.java:77)at org.mariadb.jdbc.internal.common.packet.SyncPacketFetcher.getRawPacket(SyncPacketFetcher.java:67)at org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:891)at org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:982)at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:280)- locked <0x0000000791370678> (a org.mariadb.jdbc.internal.mysql.MySQLProtocol)at org.mariadb.jdbc.MySQLPreparedStatement.execute(MySQLPreparedStatement.java:141)

Paste a part of the process stack. This process stack is actually the real stack that causes thread blocking displayed at the bottom of the tda software interface! This thread is in the runnable state, but mysql is locked. That is to say, it is blocked in mysql.