Shell multithreaded operation and thread number control instance _linux shell

Objective

In the business development process, often in the background to write some shell script processing data, but it is estimated that many people do not know that the shell script can also support multithreading, and very simple. This article is mainly about the shell implementation of multiple processes and process quantity control.

Demand

In order to better illustrate the problem, we combine examples to explain that the requirement is to scan the Url.txt file, and then determine if the URL is invalid. The content of the Url.txt file is a single line of URLs, such as:

Copy Code code as follows:

Http://www.baidu.com

http://www.google.com

Http://www.jb51.net

Single Process implementation

So the shell script scanurl.sh can write this:

Copy Code code as follows:

#!/bin/bash
#判断是否有参数
If [$#!= 1]; then
echo "The parameters you enter isn't correct!";
exit-1;
Fi

#循环读出URL并判断状态码
While Read line
Todo
{
isok= ' Curl-i-o/dev/null-s-w%{http_code} $line '
If ["$isok" = "200"]; Then
echo $line "OK"
Else
echo $line "No"
Fi
}
Done < $
echo "Execution over"

Then you can perform the following command scan:

Copy Code code as follows:

/bin/sh scanurl.sh Url.txt

However, the script executes very slowly, and 10,000 URLs are not scanned for several hours.

Multi-process implementation

It's very simple to change to a multiple process implementation, just add the & symbol to the curly brace behind do and add a wait after the done, indicating that the parent process waits for the subprocess to exit and then exits

Copy Code code as follows:

#!/bin/bash
#判断是否有参数
If [$#!= 1]; then
echo "The parameters you enter isn't correct!";
exit-1;
Fi

#循环读出URL并判断状态码
While Read line
Todo
{
isok= ' Curl-i-o/dev/null-s-w%{http_code} $line '
If ["$isok" = "200"]; Then
echo $line "OK"
Else
echo $line "No"
Fi
}
}&
Done < $
Wait
echo "Execution over"

This can be multiple concurrent execution, but there is a problem is that the process will be very much, hundreds of thousand, exceeding the system limit error, the following we add Process control.

Multi-process implementation and control number of processes

Copy Code code as follows:

#!/bin/bash
#允许的进程数
thread_num=200
#定义描述符为9的管道
MKFIFO tmp
EXEC 9<>tmp
#预先写入指定数量的换行符, a newline character represents a process
For ((i=0;i< $THREAD _num;i++))
Todo
Echo-ne "\" 1>&9
Done

If [$#!= 1]; then
echo "The parameters you enter isn't correct!";
exit-1;
Fi

While Read line
Todo
{
#进程控制
Read-u 9
{
#isok = ' Curl-i-o/dev/null-s-w%{http_code} $line '
If ["$isok" = "200"]; Then
echo $line "OK"
Else
echo $line "No"
Fi
Echo-ne "\" 1>&9
}&
}
Done < $
Wait
echo "Execution over"
RM tmp

The code above guarantees that the subprocess is in the specified number, and that the process Control principle is implemented through a pipe, which is not performed when the pipe is unreadable.

Copy Code code as follows:

{
#isok = ' Curl-i-o/dev/null-s-w%{http_code} $line '
If ["$isok" = "200"]; Then
echo $line "OK"
Else
echo $line "No"
Fi
#写入一个换行符
Echo-ne "\" 1>&9
}&

And after each process completes, a newline character is written to the pipe to ensure that the number of processes is specified.

So that we can achieve our goal.