Clientmqueue in Linux and no space left on Device

The application opened today cannot be accessed. Check the background log prompt.

2012-11-29 10:41:22:WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@11d565f -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!2012-11-29 10:41:22:WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@11d565f -- APPARENT DEADLOCK!!! Complete Status: Managed Threads: 3Active Threads: 3Active Tasks: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3f0b70 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1080f38 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1f5438 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)Pending Tasks: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@d26103com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1df3255Pool thread stack traces:Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]java.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(Unknown Source)oracle.net.ns.Packet.receive(Unknown Source)oracle.net.ns.NSProtocol.connect(Unknown Source)oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:858)oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268)oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:129)com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:862)com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:33)com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1527)com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]java.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(Unknown Source)oracle.net.ns.Packet.receive(Unknown Source)oracle.net.ns.NSProtocol.connect(Unknown Source)oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:858)oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268)oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:129)com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:862)com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:33)com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1527)com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]java.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(Unknown Source)oracle.net.ns.Packet.receive(Unknown Source)oracle.net.ns.NSProtocol.connect(Unknown Source)oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:858)oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:268)oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:129)com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:862)com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:33)com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1527)com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Then the tool connecting to Oracle cannot be opened, and the Oracle service needs to be restarted. However, after the linu command is used to restart the server, no space left on device appears]

[oracle@SERVER233 bin]$ lsnrctl startLSNRCTL for Linux: Version 10.2.0.1.0 - Production on 29-NOV-2012 10:30:23Copyright (c) 1991, 2005, Oracle.  All rights reserved.Starting /oracle10g/app/oracle/product/10.2.0/db_1/bin/tnslsnr: please wait...TNSLSNR for Linux: Version 10.2.0.1.0 - ProductionSystem parameter file is /oracle10g/app/oracle/product/10.2.0/db_1/network/admin/listener.oraLog messages written to /oracle10g/app/oracle/product/10.2.0/db_1/network/log/listener.logError listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))TNS-12549: TNS:operating system resource quota exceeded TNS-12560: TNS:protocol adapter error  TNS-00519: Operating system resource quota exceeded   Linux Error: 28: No space left on device

At first glance, it seems that the disk space is not enough, and then use the command

DF-I

Appears

[Oracle @ server233 bin] $ DF-ifilesystem inode (I) used (I) available (I) % mount point/dev/sda11 25608192 191904 25416288 1%/dev/sda1 6406144 44 6406100 1%/bootnone 214052 1 214051 1%/dev/SHM/dev/sda10 6406144 212 6405932 1%/ home/dev/sda5 25608192 19 25608173 1%/opt/dev/sda2 29949952 12 29949940 1%/sowft/dev/sda9 2562240 60 2562180 1%/tmp/dev/sda8 12812288 305651 12506637/ usr/dev/sda6 2562240 991 2561249 1%/usr/local/dev/sda7 6406144 6406144 0 100%/var [Oracle @ server233 bin] $ DF-hfilesystem capacity used available % mount point /dev/sda11 193g 17G 167g 9% // DEV/sda1 49G 91 M 46g 1%/bootnone 3.0g 0 3.0g 0%/dev/SHM/dev/sda10 49g 135 m 46g 1%/home/dev/sda5 193g 93 m 183g 1%/opt/dev/sda2 225g 731 m 213g 1%/sowft/dev/sda9 20g 77 M 19G 1%/tmp/dev/sda8 97g 6.4g 85g 7%/usr/dev/sda6 20g 177 m 19G 1%/usr/local/dev/sda7 49g 26G 21G 55%/var

The VaR directory occupies a large space and then entersClientmqueueLarge footprint found

Logs are periodically analyzed and processed. How can they occupy such a large space?

View Log File Size
# Du-SH/var/log
31 m/var/log

Hi, what file occupies a lot of space?

Continue searching.

Du-SH/var/spool/Clientmqueue/
889 M/var/spool/Clientmqueue

FindClientmqueueDirectories occupy a large amount of space.

What is the file under this directory ??

Analysis:
Some users in the system enable Cron, while the Program executed in cron has output content, and the output content will be sent to cron users in the form of mail, which is generated because Sendmail is not started;

Solution:
Add>/dev/null 2> & 1 to the command in crontab.
Note: 2> & 1: redirect the error to the place where the output is to be sent. That is, the execution result of the above command is redirected to/dev/null, that is, discard, and also discard the errors.

Rm-RF/var/spool/ClientmqueueDelete this directory.

Modify all cron

In this way, the space occupied by clinetmqueue is solved.

Reference: http://baozoutianshi.blog.sohu.com/203000776.html