發現寫Oracle的線程掛住了,情境是從mysql讀資料,然後寫到Oracle。
1 定位線程
因為在同一台機器上運行了多個java進程,要找到對應的pid,就是串連mysql的的那個進程。
[admin@arch036075.sqa.cm4 logs]$ ss -p | grep "10.232.31.67"
ESTAB 0 0 10.232.36.75:41790 10.232.31.67:mysql users:(("java",1978,184))
ESTAB 0 0 10.232.36.75:41789 10.232.31.67:mysql users:(("java",1978,179))
紅色部分就是掛住線程的pid
2 jstack
"Stage Task: mysql-to-dbsync" daemon prio=10 tid=0x00002aaac8699000 nid=0x7e33 runnable [0x0000000043cfb000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at oracle.net.ns.Packet.receive(Unknown Source) at oracle.net.ns.NSProtocol.sendReset(Unknown Source) at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1112) at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1075) at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:480) at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216) at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:966) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1170) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3339) at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3423) - locked <0x00002aaab4575880> (a oracle.jdbc.driver.T4CPreparedStatement) - locked <0x00002aaab4cda408> (a oracle.jdbc.driver.T4CConnection)
發現寫oracle的異常- locked <0x00002aaab4575880> (a oracle.jdbc.driver.T4CPreparedStatement)
- locked <0x00002aaab4cda408> (a oracle.jdbc.driver.T4CConnection)
3 詢問dba對應oracle資料庫的情況,說寫庫有一張表被鎖住了,解鎖,重啟Data Replication System系統,恢複正常。
4 之前有人說過可能是oracle的bug,設定connection的timeout可解決,也用這種方法解決了第一次遇這個的問題的應用(這是第二次遇到),重啟應用,發現是鎖表導致的。