Linux Shell file lock _linux Shell

Often in the shell script to block other processes, such as MSMTP mail queue script, this script mutex is not correct, the following is the three found in the file to achieve mutual exclusion of the correct approach.

1. The Flock of Util-linux

This command has two uses: Flock Lockfile command (FLOCK-S 200; COMMAND; 200>lockfileflock need to keep open the lock file, which is not convenient for the second use, and the-s method specifies that the file handle might conflict. The advantage is that you do not need an explicit unlock, the process exits after the lock is bound to release.

2. The Dotlockfile of Liblockfile1

Known as the most flexible and reliable file lock implementation. The wait time is related to the number of retries specified by-R, and the retry time is sum (5, ..., Min (5*n, 60), ...). The lock file does not need to be opened, and the problem is that you need to use trap exit to ensure that the lock file is deleted when the process exits.

3. The Lockfile of Procmail

Similar to Dotlockfile, but you can create multiple lock files at once.

There are two simple ways to implement file locks in a shell.

The first is to use a normal file, check the existence of a specific file when the script is started, if it exists, wait a period of time to continue checking until the file is not saved, and then delete the file at the end of the script. To ensure that the script can still be deleted when the exception exits, you can use the trap "cmd" exit TERM int command. Typically this type of file is stored in the/var/lock/directory, and the operating system cleans up the directory when it starts.
Another method is to use the flock command. The advantage of this command is to wait for the action to be done in the flock command, without adding additional code.
(Flock ... cmd flock-u) >/tmp/file.lock
But Flock has a flaw that after opening Flock fork (), the child will also have a lock, and if there is a daemon running during flock, it is necessary to ensure that daemon has closed all file handles at startup. Otherwise, the file cannot be unlocked because the daemon has been placed in the open state.

An example of Linux shell file lock implementation
Recently I've seen a lot of questions about how to keep scripts from recurring, actually the concept of file locks, and write a little example:
Starting this as a file does not result in repeated execution. (I think the names of the two scripts are exactly the same as they should not happen often)

#!/bin/bash
lockfile ()
{
 find/dev/shm/*-maxdepth 0-type l-follow-exec unlink {} \;
 [-f/dev/shm/${0##*/}]&&exit
 ln-s/proc/$$/dev/shm/${0##*/}
 trap "Exit" 0 1 2 3
}
exit ( )
{
 unlink/dev/shm/${0##*/};
 Exit 0;
}
Lockfile
# Main program
#
... #Exit

Description of the role of the/var/lock/subsys directory
Many programs need to determine whether there is currently an instance running, this directory is to let the program to determine whether there are instances of the operation of the flag, such as xinetd, if the existence of this file, indicates that there are already xinetd running , otherwise there is no, of course, the program inside also need to have the corresponding judgment measures to really determine whether there are instances in operation.
usually matching the directory with the/var/run directory, used to store the corresponding instance of the PID, if you write a script, you will find that these 2 directories together can easily determine whether many services are running, the relevant information to run and so on.
   In fact, to judge whether the file is locked is to judge it, so whether the file exists or not is a hidden lock. The contents of this directory are not necessarily locked because many services use touch to create the lock file in the startup script, and the script is responsible for clearing the lock at the end of the system, which is inherently unreliable (for example, accidental failure causes the lock file to still exist), I am in the script is generally combined with the PID file (if there is a PID file), get the PID of the instance from the PID file, then use PS to test whether the PID exists, so as to determine whether the actual instance is running, the more secure way is to communicate with the process, but in this case the script alone can not do it.
 
Flock command belongs to the UTIL-LINUX-2.13-0.46.FC6 package on my system, and if not, try to update the Util-linux package under your system.
reason for this command:
The discussion of scripting serialized by Brother Woodie in the forum has been perfected.
But flock this command combines well with shell scripts and is very similar to the Flock function usage in languages such as c/perl/php, and is simple to use. By contrast, Woodie's content needs a shallow shell foundation to understand. The
Two formats are:
       flock [-sxon] [-w timeout] lockfile [-c] command ...

Flock [-sxun] [-w timeout] fd
To introduce the parameters:
-S is a shared lock, a request to set an exclusive lock on an FD directed to a file has failed while another process attempts to set a shared lock on the FD that is directed to the file, and other processes have attempted to do so.
-E is an exclusive or exclusive lock, and during the time that an exclusive lock is set on an FD directed to a file without releasing the lock, the other process attempts to set a shared or exclusive lock on the FD that is directed to the file to fail. This parameter is set by default as long as the-s parameter is not set.
-U manually unlock, generally do not have to, when the FD shutdown, the system will automatically unlock, this parameter used in the script command part of the need for asynchronous execution, part of the situation can be synchronously executed.
-N is non-blocking, when an attempt to set the lock fails, in non-blocking mode, returns 1 directly, and continues with the following statement.
-W Sets the blocking timeout and, when the number of seconds exceeds the set, jumps out of the block, returns 1, and continues with the following statement.
-O must be available when the first format is used, which means to turn off the FD of the set lock before executing the command so that the child process of the command does not remain locked.
-C executes the subsequent comand.
for a practical example:

#!/bin/bash
{
flock-n 3
[$-eq 1] && {echo fail; exit;}
echo $$ sleep
} 3<>mylockfile

The function of this example is that when one instance of a script is executing, another process that attempts to execute the script fails to exit.
The sleep sentence can be replaced with the statement segment you want to execute.
Note here that I use <> to open mylockfile because the directional file descriptor is executed before the command. So if you need to read and write the Mylockfile file in the statement segment you want to execute, for example, want to get the PID of the previous script instance, and write the PID of this script instance to mylockfile. At this point, the direct use of > Open mylockfile will empty the last deposited content, and < open mylockfile when it does not exist will cause an error. Of course these problems can be solved in other ways, I just point out the most common method.

"Background Information"

There were several posts on the CU that discussed a practical problem, that is, how to limit the time allowed to only one script instance to run. This version of the new and old owner and other netizens have participated in the discussion, but to Faintblue brother's post for everyone to inspire the most, the following background of many of the content is from him. Thank Faintblue here, also thank the owner and other friends!
Woodie summed up the existing results, generally can be divided into two ways:
One simple way is to use the PS Class command to find the number of scripts that have been run, if greater than or equal to 2 (don't forget to include yourself in the ^_^), exit the current script, equal to 1, then run. This approach is simple, but there are some questions:
First of all, PS get the number of script file processes there are many traps, such as sometimes can not PS to the name of the script file;
Even if you can PS to the script name, if used in the pipeline, because of the child shell, in most of the platform will get strange results, and sometimes get the number a, sometimes get the number B, let a person at a loss;
Even if the counting problem has been solved, there is a problem, but not too serious: If two script instances are counted at the same time, it is clear that the numbers should equal 2, and two are out. So no script is executing at this point in time;

Second, the method of adding locks. That is, the script tries to get a "lock" at the start of execution, and then it goes ahead and quits.
There are also some problems with the method of locking, which mainly focus on two aspects:
First, how to avoid the race condition (race condition) when lock. That is, how to find some "atomic" operations, so that the action of the lock is completed one step, the middle can not be interrupted. Otherwise, the following conditions may occur:
Script 1 detects that no locks are occupied;
Then script 2 also detects that no locks are occupied;
Script 1 plus lock, start execution;
Then script 2 (incorrectly) lock, also start execution;
See, two scripts are executed at the same time. :(
Some of the possible locking "atomic" operations are:
1. Create the directory, when a process is successfully created, other processes will fail;
2. Symbolic Link: ln-s, a link creation after the other process ln-s command will be wrong;
3. Competition in the first row of the file, multiple processes are written to the file at the same time in a append way, only one by one processes are written to the first line of the file, because there is no way to have two first row. ^_^
4. Other software package lock tools, usually the C language binary program, you write the line.
At present, the problem of lock can be solved.
Second, find a way to avoid the "deadlock" situation, which means that although the "lock" is occupied, no script is executing. This usually happens after the script quits unexpectedly, after it is too late to release the occupied lock. If you receive some system signals to exit, the machine accidentally fell out of power and so on.
In the case of the former, a trap can be used to capture some signals and release the lock before exiting, but some signals cannot be captured.
For the latter, you can resolve the lock with a script automatically after the machine is reset. But a little trouble.
So it's ideal for the script to detect the deadlock itself and then release it. But the problem is how to find an "atom" operation that will detect the deadlock and delete the deadlock in one step, or else it will appear the same as the lock when the problem of competing conditions. For example:
The deadlock was detected by process 1;
Process 2 monitor to deadlock;
Process 1 Deletes the deadlock;
Process X (may also be process 1 itself) lock, start to run;
Process 2 (incorrectly) delete deadlock;
The lock is not occupied at this time, so any process can be locked and put into operation.
This shows the situation where two processes are running concurrently. :(
Unfortunately: after the discussion so far, Woodie has not found a suitable "atomic" operation. :(just found a slightly better way: in the deletion of the file with the Inode for identification, so the other process new locks (although the file name is the same, but the probability of the same inode is relatively small) is not easy to accidentally deleted. This method is close to perfect, unfortunately there is a small chance of accidentally deleted, can not be said to be 100% security. Alas, Shan! :(

Recently, some netizens asked this question, prompting me to think again. From one of my previous ideas developed a bit, change a way of thinking, there will be enlightened feeling. Do not dare to hide the private, write to please debug. ^_^

The basic idea is: learn from the concept of the critical area in the process of programming, if each process into the critical area we set up, only one by one in order to enter, can not guarantee that only one script at a time to run it? How to establish such a critical zone? I thought of a way to use a pipe, multiple processes to write to the same pipe, can only be entered in one line, the other end is also a line of read out, so that the parallel execution of multiple processes into the critical area of the "serialization." This is similar to the way that brother Faintblue used to post append documents.
We can allow parallel processes to write one line of requests to a single pipe at a time. The content is its process number, which is read sequentially at the other end of the pipe, but only the first request gets a "token", is allowed to start, and subsequent requests are ignored, and the corresponding process exits without a token. This ensures that only one process is running at any time (strictly speaking, entering the critical section). When it comes to "tokens," friends familiar with the history of the Web may associate IBM's Token Ring architecture, where only one host gets the token and sends the data, without the "collision" problem of Ethernet. Unfortunately, as with micro-channel technology, IBM's technology is good, but in the end it was eliminated. Yes, the concept of a token here is borrowed for the token ring. ^_^
When a process completes, sends a termination signal to the pipe, returns "token", the other end receives it, and begins to select the next process to issue "token".
You might ask, so how does a deadlock problem work? Don't worry, I have in the previous discussion has proposed the detection processing deadlock the code to separate out, to a specialized process to deal with the idea, here on the specific practice of such a thinking. When the task of detecting and deleting deadlocks is performed by a specialized process, there are no multiple concurrent processes operating on the same lock, so the material basis for the occurrence of a competing condition does not exist at all. ^_^
How about developing this idea and allowing simultaneous execution of multiple processes? Of course! As long as a counter is set up, the number that reaches the limit stops issuing "token".
The following is a woodie of the above ideas, just under the CentOS 4.2 simple test, there may be a lot of mistakes, please help "to remove worms." ^_^ The idea of what the problem also please feel free:
Script 1,token.sh, responsible for token management and deadlock detection processing. As with the next script, to preserve the maximum compatibility of the script, use the syntax of the Bourne shell as much as possible, using printf instead of the echo,sed usage and keeping it as generic as possible. Here is a named pipe that accepts the request, and the token is emitted in a file. If you use Ksh may use the process to achieve, familiar with ksh friends can try. ^_^

#!/bin/sh #name: token.sh #function: Serialized token distribution, at anytime, only a cerntern number of token given O UT #usage: token.sh [number] & #number is set to allow number of scripts to run in same time #if no # is given , the default value is 1 if [-p/tmp/p-aquire]; Then rm-f/tmp/p-aquire fi if mkfifo/tmp/p-aquire; Then printf "Pipe File/tmp/p-aquire created\n >>token.log Else printf" Cannot create pipe file/tmp/p-aquire\ N ">>token.log exit 1 fi loop_times_before_check=100 if [-N" $ "];then limit=$1 Else # default Concurre 
 NCE is 1 limit=1 fi number_of_running=0 counter=0 while:;d o #check stale token, which the owner is died unexpected If ["$counter"-eq "$loop _times_before_check"]; Then counter=0 for PID in ' Cat Token_file ';d o pgrep $pid if [$?-ne 0]; Then #remove lock printf "s/$pid//\nwq\n" |ed-s token_file number_of_running= ' expr $number _of_running 
-1 ' fi done fi Counter= ' expr $counter + 1 ' # if ["$number _of_running"-ge "$limit"];then # token are all given out. 
  Bypass all request until a instance to give one back pid= ' Sed-n '/stop/{s/\ ([0-9]\+\) \+stop/\1/p;q} '/tmp/p-aquire ' If [-N "$pid"]; Then # get a token returned printf "s/$pid//\nwq\n" |ed-s token_file number_of_running= ' expr $number _of_runni Ng-1 ' Continue fi else # there is still some token to give out. Serve another request read PID action </tmp/p-aquire if ["$action" = stop]; 
     Then # One token is given. printf "s/$pid//\nwq\n" |ed-s token_file number_of_running= ' expr $number _of_running-1 ' Else # it ' s a r Equest, give off a token to instance identified by $pid printf "$pid" >> token_file number_of_running= ' Expr $number _of_running + 1 ' fi fi done

--------------------------------------------------------------------------------------------
Revision history:
1. Fix a bug in token.sh, replace the original command with SED to remove the invalidation token with the ED command. Thanks to r2007 and Waker two-bit pointing out the error!
--------------------------------------------------------------------------------------------

Script 2: Concurrent execution of the script--my-script. Insert your own code after the "Your code goes Here" line, which I used to test.

#!/bin/sh 
# Second to ' Wait ' ditributer gives off a token 
a_while=1 
if [!-p/tmp/p-aquire]; then
   
    printf "Cannot find file/tmp/p-aquire\n" >&2 
 exit 1 
fi 
# try to aquire a token 
printf "$$\n" >  >/tmp/p-aquire sleep 
$a _while 
# Click If we get one 
grep "$$" Token_file 
if [$-ne 0]; then 
 # Bad luck. :( 
 printf "No token free now, exitting...\n" >&2 
 exit 2 
fi
   

This script is to lock the file, but I have some doubts about the script, for the time being, not to discuss, and then later come back to talk about

#!/bin/sh # filelock-a Flexible file locking mechanism.  retries= "#" # Default number of retries action= "lock" # Default action nullcmd= "/bin/true" # NULL command for Lockfile while getopts "Lur:" OPT;
  Do case $opt in L) action= "lock";
  u) action= "unlock";;
 R) retries= "$OPTARG";; ESAC Done Shift $ (($OPTIND-1)) if [$#-eq 0]; Then cat << EOF >&2 Usage: $ [-l|-u] [-R Retries] Lockfilename where-l The requests a lock (the default),-u r
Equests an unlock,-R X specifies a maximum number of retries before it fails (default = $retries). EOF Exit 1 Fi # ascertain whether we have LOCKF or lockfile system apps if [-Z ' $ (which Lockfile | Grep-v ' ^no ') "]; Then echo "$ failed: ' lockfile ' utility not found in PATH." >&2 Exit 1 fi if ["$action" = "lock"]; Then if! Lockfile-1-R $retries "$" 2>/dev/null; Then echo "$0:failed:couldn ' t create Lockfile in Time" >&2 exit 1 fi else # action = Unlock if [!-F "$"]; Then echo "$0:warning:lockfile $ doesn ' t exist to unlock" >&2 exit 1 fi rm-f "$" fi exit 0