O & M Case Study: "argument list too long" error and Solution

As an O & M engineer, this error is not unfamiliar. When executing commands such as RM, CP, and MV, if the number of files to be operated is large, wildcards may be used to batch process a large number of files, in this case, the problem of "argument list too long" may occur.

1. Error

This is a MySQL database server that runs many scheduled tasks in the system. Today, another scheduled task is added through the crontab command. The following error occurs when you exit:

#crontab -e

After editing, save and exit. the following error is displayed:

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M00/38/1E/wKioL1OzZfWDqd19AABapEtvGzQ465.jpg "Title =" 22.png" alt = "wkiol1ozzfwdqd19aabapetvgzq465.jpg"/>

 

2. Solution

According to the error message above, it is basically determined that the disk space is full. First, check the disk space of the server and check the/tmp disk space according to the error message, then it should be the disk space of the root partition, preferably the disk space of other partitions of the system.

3. troubleshooting

Run the DF command to check all the disk partitions on the server. There is still a lot of/tmp partition space, and there is still a lot of space available in the root partition, the final result is that/var has used 100% of the disk partition space, and the problem has been located here. It is caused by the full/var disk space, because crontab writes the file information to the/var directory when saving the disk, the error is taken for granted because the disk has no space.

4. Solve the Problem

Run the "Du-sh" command to check the size of all files or directories in the/var directory. It is found that the/var/spool/clientmqueue directory occupies 90% of the total partition size of/var, so how are the files in the/var/spool/clientmqueue directory generated? Can they be deleted? The following describes how to generate files in the/var/spool/clientmqueue directory.

You can open some files in the/var/spool/clientmqueue directory to check whether there are some mail Information. Most of the mail content is about cron daemon, in fact,/var/spool/clientmqueue is a directory for temporary email storage. By default, the Linux server sends some emails. For example, when the Program executed by cron has output content, the system will send the mail information to the user who executes the cron process. When sending the mail, the system will first copy the mail to the/var/spool/clientmqueue directory, then, wait for the MTA Program (mail transfer agent) to process the email. The main function of MTA is to transfer the email in this directory to the/var/spool/mqueue directory, and then send it to the real destination through the sendmail service. So I checked the sendmail service on this server and found that it was not enabled. The reason for the large/var/spool/clientmqueue directory was found: the client service that did not send emails, all emails are accumulated in this directory.

After confirming that the content is useless, switch to the/var/spool/clientmqueue directory and run the RM command to delete all files. An error occurs:

[[email protected] clientmqueue]# rm */bin/rm: argument list too long

At this point, we are talking about the problem at the beginning of this article.

When Linux tries to pass too many parameters to a system command, the "argument list too long" error will occur. This is a constant limitation in Linux. To view this restriction, run the "getconf arg_max" command, as shown in:

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M01/38/1F/wKiom1OzZnzwrhPoAABfkvbQ0a8570.jpg "Title =" 33.png" alt = "wkiom1ozznzwrhpoaabfkvbq0a8570.jpg"/>

This is the maximum value of centos6, and in centos5.x, this value is relatively small, as shown in:

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M01/38/22/wKiom1OzcT2SsMYVAABy410ycC0222.jpg "Title =" 44.png" alt = "wkiom1ozct2ssmyvaabypolicycc0222.jpg"/>

Therefore, this issue occurs in earlier Linux versions.

After knowing the cause of the problem, there are many solutions. Here we provide four solutions to this problem, which are described as follows:

(1) manually divide command line parameters into smaller parts

For example:

rm [a-n]* -rfrm [o-z]* -rf

This method is the simplest, but relatively mentally retarded, because you must know how to evenly split the file. In the case of a large number of files, you need to enter commands many times.

(2) Use the find command to delete

The basic principle is to use the find command to filter the file list and pass the files that meet the requirements to a series of commands. This method is the most concise and effective.

For example:

find /var/spool/clientmqueue -type f -print -exec rm -f {} \;

However, this method also has a disadvantage: It is necessary to traverse all files, so it is time-consuming to consume a large number of files.

(3) Use shell scripts

This method is implemented by writing a shell script and then using a sequential statement, similar to the find method.

For example, you can write the following script:

#! /Bin/bash # Set the folder to be deleted

RM_DIR=‘/var/spool/clientmqueue‘cd $RM_DIRfor I in `ls`do rm -f $Idone

(4) re-compile the Linux Kernel

In this method, you need to manually add the number of pages allocated to the command line parameters in the kernel, open the include/Linux/binfmts. h file under the kernel source, and find the following lines:

# define MAX_ARG_PAGES  32

Change "32" to a larger value, such as 64 or 128, and then recompile the kernel.

This method is permanently effective and can completely solve the problem, but it is complicated and recommended to advanced users. Users without Linux experience are not recommended to use this method.

This article from the "Technical Achievement dream" blog, please be sure to keep this source http://ixdba.blog.51cto.com/2895551/1433379