File System/var space 100% Problems

File System/var space 100% Problems

 

I. Problem Description

Multiple df headers appear in the/var/spool/mqueue directory, resulting in the final 100%/var Space

 

EBANK_P570_MAIN/var/spool/mqueue#ls -ltotal 8732656-rw-r--r--    1root     system          571 Aug 20 2012  Qfq7JGw5K21643126-rw-r--r--    1root     system          569 Aug 23 2012  Qfq7MGvxch987418-rw-r--r--    1root     system          572 Jun 25 02:00 Qfs5OHxF6d1339552-rw-r--r--    1root     system          570 Jul 04 01:59 Qfs63HwefO1359972-rw-r--r--    1root     system          773 Jul 04 02:02 Qfs63I1DfN1790554-rw-r--r--    1root     system          570 Jul 06 02:00 Qfs65Hx5Vq1315640-rw-r--r--    1root     system          773 Jul 06 02:02 Qfs65I16Vp1483544-rw-------    1root     system          760 Jul 06 02:39 Qfs65IaKdR1397376-rw-r--r--    1root     system          573 Jul 07 00:58 Qfs66GvJMf1332120-rw-r--r--    1root     system            0 Aug 23 2012  dfq7JGw5K21643126-rw-r--r--    1root     system            0 Aug 23 2012  dfq7MGvxch987418-rw-r--r--    1root     system   1153011528 Jul 06 02:00 dfs65Hx5Vq1315640-rw-r--r--    1root     system   1153012885 Jul 06 02:02 dfs65I16Vp1483544-rw-------    1root     system   1153012885 Jul 06 02:39 dfs65IaKdR1397376-rw-r--r--    1root     system   1011801454 Jul 07 00:58 dfs66GvJMf1332120 

 

Ii. Cause Analysis:

1. Query Information:The/var/spool/mqueue directory contains temporary files associated with each message in the queue. That is, the temporary directory for storing emails before sending emails. The output results of each scheduled task are sent to the user by email. The mail queue directory is/var/spool/mqueue, the mail queue contains four types of mail queue files: data files, control files, temporary files, and copy files. The file format is as follows:

DfAA00312 Data File

QfAA00312 Control File

TfAA00312 temporary file

XfAA00312 copy this file

 

2. To further view the file content, the crontab executes the/ora_backup/rep/repsj/imp. sh script to produce a large mail, resulting in full/var space.

EBANK_P570_MAIN/var/spool/mqueue#more dfs66HxcE51216754Cron Environment: SHELL =/usr/bin/sh PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java14/jre/bin:/usr/java14/bin CRONDIR=/var/spool/cron/crontabs ATDIR=/var/spool/cron/atjobs LOGNAME=oracle HOME=/oracle Your "cron" job executed on EBANK_P570_MAIN onMon Jul  7 00:40:00 BEIST 201 exportBATCH_AUTO_RUN=Y; . ./.profile; sh /ora_backup/rep/repsj/imp.sh produced the following output:2014042320140706。。。。。

 

3. Confirm the cause: Because the Script output in crontab is too large, the mail is too large and the/var space is full. The solution is as follows:

A. The output of the script is redirected to/dev/null. Because the script is no longer output, no email is sent.

* ***** A. sh>/dev/null 2> & 1.

B. Add MAILTO = "" to the front of crontab ""

$ Crontab-e

At the top of the file, enter:

MAILTO = ""

 

4. Confirm the cause by monitoring the system trend chart.

The utilization diagram of the latest system/var shows that,

1) recently, the utilization rate of/var space has been growing slowly, with a sudden increase around June 26, but a 100% problem occurred recently.

2) there is actually a utilization peak every night, but at six o'clock, the utilization rate is reduced to normal, because the six o'clock scheduled task cleared the mail of oracle users.

EBANK_P570_MAIN/# crontab-l oracle

0 6 * cat/dev/null>/var/spool/mail/oracle

3) Why does a large file appear in the/var/spool/mqueue directory?

Through testing on the testing machine,

A. the output produced by crontab is first placed in the/tmp directory (corresponding to Figure 2/tmp0: 30 and the space usage is continuously increasing)

B. After the crontab command is executed, the mv is sent to the/var/spool/mqueue directory to send an email to the oracle user. (the/var space Suddenly increases)

C. It is normally accepted by oracle users and stored in the/var/spool/mail/oracle file.

D. When scheduled tasks are executed at on the oracle mail, the/var corresponding to Figure 1 suddenly drops

 

4. Why is there no problem in the early morning?

Due to the thorough cleaning of/var yesterday, the mqueue directory can save the emails to be sent, increase the space from 9% to 42%, and then be cleared at six o'clock.

Figure 1:/var space utilization

 

Figure 2/tmp File System Utilization

Figure 3. Today/var Utilization

 

Iii. Solutions

1. temporary solution:

Quick cleanup solution: Transfer big data files to the/backup/directory

Mv/var/spool/mqueue/df */backup/

 

2. permanent solution. Therefore, we recommend that you rewrite the script writing, especially the execution script with a large number of output results.

The script is redirected to/dev/null to prevent large log files.