A business failure caused by the full inode node

A business failure caused by the full inode node

http://2358205.blog.51cto.com/2348205/1747951

Long time no blog post, today in Saturday, share the just finished a glitch

Description of the phenomenon:

Operating sister paper side response operation background error, as follows:

At first thought is TMP directory does not have permission to write, view directory permissions, 777, not the problem;

View Nginx error log, some error message is as follows, 500 error:

113.xx.xx.48--[05/mar/2016:19:33:09 +0800] "Post/index.php?r=site/login http/1.1" 112266 Index.php?r=site/login "" mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) applewebkit/537.36 (khtml, like Gecko) chrome/48.0.2564.116 safari/537.36 "

14.xx.xx.67--[05/mar/2016:19:33:14 +0800] "Get/index.php?r=site/login http/1.1" 120922 "-" "mozilla/5.0 (Windows N T 6.1) applewebkit/537.36 (khtml, like Gecko) chrome/38.0.2125.122 safari/537.36 SE 2.X METASR 1.0 "

Check the MySQL error log, the log is as follows, but view earlier, there are the following error, before the problem, why this time error, should not be caused by this problem, continue to check ...

Log into the database, view the database table, you can select the table data, but the DESC structure, successive desc a few tables, the results are the same, reported the following error:

Is the table damaged? The execution of the table Repair command, the command is as follows:

/usr/local/mysql/bin/mysqlcheck--all-databases-uusername-ppassword-r

Some of the table repair procedures are as follows:

After the repair is complete, the Mother egg, or the same error message,

Take a closer look, mother-of-the-egg, does this mean you can't write temporary files? But why does DESC involve the creation of temporary files? Need to dig deep!!

ERROR 1 (HY000): Can ' t create/write to file '/tmp/#sql_3b25_0. MYI ' (errcode:28)

is the disk space full? The alarm information is confiscated ... With a confused mind to view the next disk space, as follows:

[Email protected] ~]# df-h

Filesystem Size used Avail use% mounted on

/dev/xx1 20G 6.9G 12G 37%/

Tmpfs 3.9G 0 3.9G 0%/dev/shm

/dev/xxx 63G 26G 34G 44%/mnt

Not full. Strange.. See if there are any changes to the my.cnf file.

Vim has reported a mistake. As follows:

e303:unable to open swap file for "my.cnf", Recovery impossible

At first did not notice, continue to edit, read the contents of the file, did not change Ah, the authority and the owner is also normal, it is strange.

And then vim a bit other files, there is also a hint error .... Unable to create interchange file:

So I try to touch, alas, can't create new files, mkdir, is the same. Prompt for no disk space ...

Touch:cannot Touch ' e ': No space left on device

Just Df-h is not obvious there is room. is the node of the file full? Decisive df-i ...

[Email protected] ~]# df-i

Filesystem inodes iused IFree iuse% mounted on

/dev/xxx 1310720 171826 1138894 100%/

TMPFS 1007261 1 1007260 1%/DEV/SHM

/dev/xxx 4194304 463685 3730619 12%/mnt

Sure enough,/directory of the number of nodes used 100%, so, the problem found, then check what exactly is what the file occupies so many file nodes.

Finally, there are a lot of small files located under the/var/spool/clientmqueue.

Description: The program that Cron executes in the system has the output content, the output content will send to the cron user in the message form, and the SendMail does not start, therefore produces these files;

Now that you know the cause of this file, delete the chant, the current business to ensure the normal operation of the most important.

CD to/var/spool/clientmqueue this directory, use RM-RF./*, oops, not to delete? Error:

/bin/rm:argument list too long

Finally use LS |xargs rm-rf delete ....

Note: The parameters that are received by default with RM are limited, and you can use this command to output file names and delete them in groups.

With Df-i, the number of nodes in the/directory is also freed up. Visit the business again and get it done!!

[Email protected] clientmqueue]# df-i

Filesystem inodes iused IFree iuse% mounted on

/dev/xxx 1310720 171826 1138894 14%/

TMPFS 1007261 1 1007260 1%/DEV/SHM

/dev/xxx 4194304 463685 3730619 12%/mnt

Description: Inode translated into Chinese is the index node, each storage device (such as a hard disk) or storage device partition is formatted as a file system, there should be two parts, part of the inode, and the other part of the Block,block is used to store data. The inode is the information that is used to store this data, including file size, owner, attribution user group, read-write permission, and so on. The Inode indexes the information for each file, so there is a value for the inode. According to the instructions, the operating system can find the corresponding file by the Inode value.

Monitoring System In addition to the remaining disk capacity monitoring, the number of remaining inode should be monitored also add ~
Df.bytes.free.percent; Df.inodes.free.percent

Reasons for large number of files in/var/spool/clientmqueue/directory and Solutions 2
http://blog.itpub.net/22036495/viewspace-1056821/

Problem phenomenon: A large number of files exist in the/var/spool/clientmqueue/directory in the Linux operating system .

Cause Analysis:

The system has a user to open cron, and cron execution of the program has output content, the output will be sent to the user cron mail, and SendMail did not start so that the resulting files;

Workaround:
1, the crontab inside the command after adding >/dev/null 2>&1

A business failure caused by the full inode node