今日開啟程式發現無法訪問,一看後台日誌提示
2012-11-29 10:41:22:WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@11d565f -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!2012-11-29 10:41:22:WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@11d565f -- APPARENT DEADLOCK!!! Complete Status: Managed Threads: 3Active Threads: 3Active Tasks: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3f0b70 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1080f38 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1f5438 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)Pending Tasks: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@d26103com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1df3255Pool thread stack traces:Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]java.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(Unknown Source)oracle.net.ns.Packet.receive(Unknown Source)oracle.net.ns.NSProtocol.connect(Unknown Source)oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:858)oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268)oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:129)com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:862)com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:33)com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1527)com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]java.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(Unknown Source)oracle.net.ns.Packet.receive(Unknown Source)oracle.net.ns.NSProtocol.connect(Unknown Source)oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:858)oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268)oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:129)com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:862)com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:33)com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1527)com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]java.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(Unknown Source)oracle.net.ns.Packet.receive(Unknown Source)oracle.net.ns.NSProtocol.connect(Unknown Source)oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:858)oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268)oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:129)com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:862)com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:33)com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1527)com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
然後串連oracle的工具也打不開,感覺需要重啟oracle服務,但是去伺服器採用Linu命令重啟後就出現【No space left on device 】
[oracle@SERVER233 bin]$ lsnrctl startLSNRCTL for Linux: Version 10.2.0.1.0 - Production on 29-NOV-2012 10:30:23Copyright (c) 1991, 2005, Oracle. All rights reserved.Starting /oracle10g/app/oracle/product/10.2.0/db_1/bin/tnslsnr: please wait...TNSLSNR for Linux: Version 10.2.0.1.0 - ProductionSystem parameter file is /oracle10g/app/oracle/product/10.2.0/db_1/network/admin/listener.oraLog messages written to /oracle10g/app/oracle/product/10.2.0/db_1/network/log/listener.logError listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))TNS-12549: TNS:operating system resource quota exceeded TNS-12560: TNS:protocol adapter error TNS-00519: Operating system resource quota exceeded Linux Error: 28: No space left on device
咋一看貌似,磁碟空間不夠,然後採用命令
df -i
出現
[oracle@SERVER233 bin]$ df -iFilesystem Inode (I)已用 (I)可用 (I)已用% 掛載點/dev/sda11 25608192 191904 25416288 1% //dev/sda1 6406144 44 6406100 1% /bootnone 214052 1 214051 1% /dev/shm/dev/sda10 6406144 212 6405932 1% /home/dev/sda5 25608192 19 25608173 1% /opt/dev/sda2 29949952 12 29949940 1% /sowft/dev/sda9 2562240 60 2562180 1% /tmp/dev/sda8 12812288 305651 12506637 3% /usr/dev/sda6 2562240 991 2561249 1% /usr/local/dev/sda7 6406144 6406144 0 100% /var[oracle@SERVER233 bin]$ df -hFilesystem 容量 已用 可用 已用% 掛載點/dev/sda11 193G 17G 167G 9% //dev/sda1 49G 91M 46G 1% /bootnone 3.0G 0 3.0G 0% /dev/shm/dev/sda10 49G 135M 46G 1% /home/dev/sda5 193G 93M 183G 1% /opt/dev/sda2 225G 731M 213G 1% /sowft/dev/sda9 20G 77M 19G 1% /tmp/dev/sda8 97G 6.4G 85G 7% /usr/dev/sda6 20G 177M 19G 1% /usr/local/dev/sda7 49G 26G 21G 55% /var
覺得是var目錄佔用空間比較大 然後進入clientmqueue 發現佔用比較大
日誌是定期分析處理的,怎麼會佔這麼大的空間?
查看記錄檔大小
# du -sh /var/log
31M /var/log
好小耶,那到底是什麼檔案佔用了大量空間了?
繼續找。
du -sh /var/spool/clientmqueue/
889M /var/spool/clientmqueue
找到clientmqueue 目錄佔了大量空間。
這個目錄底下的檔案到底是幹嘛的??
分析:
系統中有使用者開啟了cron,而cron中執行的程式有輸出內容,輸出內容會以郵件形式發給cron的使用者,而sendmail沒有啟動所以就產生了這些檔案;
解決辦法:
將crontab裡面的命令後面加上> /dev/null 2>&1
註:2>&1:把錯誤重新導向到輸出要送到的地方。即把上述命令的執行結果重新導向到/dev/null,即拋棄,同時,把產生的錯誤也拋棄。
rm -rf /var/spool/clientmqueue 刪除這個目錄。
重新修改所有的cron
這樣的話就解決了clinetmqueue佔用空間的問題。
參考網址:http://baozoutianshi.blog.sohu.com/203000776.html