Bash's multi-threading model

Shell Multithreading Support
one, single-threaded shell
Cat test.html

http://blog.51cto.com/hmtk520/1943716http://blog.51cto.com/hmtk520/ 1944156http://blog.51cto.com/hmtk520/1944015http://blog.51cto.com/hmtk520/1944004http://blog.51cto.com/hmtk520 /1943976http://blog.51cto.com/hmtk520/1943953http://blog.51cto.com/hmtk520/1967968http://blog.51cto.com/ Hmtk520/1950172http://blog.51cto.com/hmtk520/1950170http://blog.51cto.com/hmtk520/1950157http://blog.51cto.com /hmtk520/1950148http://blog.51cto.com/hmtk520/1942867http://blog.51cto.com/hmtk520/1942866http:// blog.51cto.com/hmtk520/1941762http://blog.51cto.com/hmtk520/1941755http://blog.51cto.com/hmtk520/1941732http:/ /blog.51cto.com/hmtk520/1941721http://blog.51cto.com/hmtk520/1941652http://blog.51cto.com/hmtk520/1941650http ://blog.51cto.com/hmtk520/1941644 

[Email protected] test]# sum=0; time while read line; Do curl-s $line &>/dev/null && let sum++;d one < test.html; Echo $sumreal 0m5.960suser 0m0.113ss Ys 0M0.158S20

If you use the Curl command directly, you need more than 5s

[[Email protected] test]# time for I in ' cat test.html ';d o curl-s $i &>/dev/null;d onereal 0m5.386suser 0m 0.122ssys 0m0.138s

Serial execution is less efficient and slower.

Second, upgrade the use of multithreading

[[email protected] test]# time ./thread.html  test.html ok... okreal    0m0.594suser    0m0.067ssys     0m0.113s//takes less than 1s [[email protected] test]# cat thread.html #!/bin/ bashwhile read line do {    curl -s  $line  &>  /dev/null && echo ok} &done < $1wait //waits for all threads to exit after execution is complete. Exits when the first thread finishes executing. [[email protected] test]# cat thread.html  //This can also #!/bin/bashfor i in   ' cat ./test.html '  ; do{    curl -s  $i  &> / Dev/null && echo ok} &done wait 

Using & and wait

Three, multithreading issues
But one problem is that the process will suddenly be very much, hundreds of thousand, exceeding the system limit error, need to control the number of processes.

#!/bin/bashfile=./index.htmlthread_num=50 #  custom concurrency number, Depending on the server performance or application capacity sizing, do not define too large, too large may cause system downtime tmp_fifo_file=/tmp/$$.fifo  #以进程ID号命名的管道文件mkfifo   $tmp _fifo_file   #创建临时管道文件exec  4<> $tmp _fifo_file  #以rw方式打开tmp_fifo_file管道文件, the file descriptor is 4, you can also take 3-9 of any descriptor rm - f  $tmp _fifo_file  #删除临时管道文件, or do not remove for  (i=0;i< $thread _num;i++)   #利用for循环向管道中输入并发数量的空行do     echo  ""   #输出空行done  >&4  #输出重定向到定义的文件描述符4上for  i in   ' cat  $file '; do    read -u4  #从管道中读取行, each row, all rows are suspended after reading until the pipeline has idle rows     {    curl -s  $i  &> /dev/null & & echo  $i      sleep 0.1  #暂停0.1s to system buffering time, limit number of concurrent processes      echo  ""  >&4    }&  #放入后台运行donewait  # Waits for all background processes to finish executing exit 0 

[[Email protected] test]# cat bingfa.sh #!/bin/bashfile=./index.htmlthread_num=50  #  custom concurrency, resizing based on server performance or applicability, not defined too large, too large may cause system downtime tmp_fifo_file=/tmp/$$.fifo  #以进程ID号命名的管道文件mkfifo   $tmp _fifo_file  #创建临时管道文件exec  4<> $tmp _fifo_file  #以rw方式打开tmp_fifo_file管道文件, file descriptor 4, You can also take 3-9 of any descriptor rm -f  $tmp _fifo_file  #删除临时管道文件, or you can delete for  (i=0;i< $thread _num;i++)  # Use a For loop to enter a concurrent number of empty lines into the pipeline do    echo  ""   #输出空行done  >&4 # Output redirected to the defined file descriptor 4 on Function curl_url ()  {for i in  ' cat  $file '; do     read -u4  #从管道中读取行, one line at a time, after all rows have been read and suspended until the pipeline has an idle line     {     curl -s  $i  &> /dev/null      sleep 0.1   #暂停0.1s to the system buffer time to limit the number of concurrent processes     echo  "&NBSP;&GT;&AMP;4&NBSP;&NBSP;&NBSP;&NBSP;} &  #放入后台运行donewait   #等待所有后台进程执行完成}for j in {1..200}; do    curl_url && echo  "$j  times succeed " doneexit 0[[email protected] test]# while read  line ; do echo  $line  ; done < index.html  [[email  protected] test]# while read line ; do  curl -s  $line   &> /dev/null && echo finished   ; done <  index.html //can run normally [[Email protected] test]# while read line ; do  { curl -s  $line  &> /dev/null && echo finished  } ; done < index.html //this is not working.

Bash's multi-threading model