Linux concurrency implementation

1. Non-concurrent execution

[Root @ xml-ora1 ~] # Cat nopara. Sh

#! /Bin/sh

For (I = 0; I <5; I ++); do

{

Echo "$! Start 'date '.";

Sleep 5;

Echo 1> AA & Echo "$! Done 'date '.";

}

Done

Wait

Cat AA | WC-l

Rm AA

[Root @ xml-ora1 ~] # Sh nopara. Sh

Start mon mar 18 16:50:25 CST 2013.

Donemon mar 18 16:50:30 CST 2013.

Start mon mar 18 16:50:30 CST 2013.

Donemon mar 18 16:50:35 CST 2013.

Start mon mar 18 16:50:35 CST 2013.

Donemon mar 18 16:50:40 CST 2013.

Start mon mar 18 16:50:40 CST 2013.

Donemon mar 18 16:50:45 CST 2013.

Startmon mar 18 16:50:45 CST 2013.

Donemon mar 18 16:50:50 CST 2013.

5

 

[Root @ xml-ora1 ~] # Ps-Ef | grep para | grep-V grep

Root 19294 3092 0 00:00:00 pts/2 Sh nopara. Sh

Ii. Concurrent execution

In bash, background tasks are used to implement multi-process tasks ". In this case, a background execution & symbol is added on the basis above. At this time, five cyclic tasks should be executed concurrently.

[Root @ xml-ora1 ~] # Cat para. Sh

#! /Bin/sh

For (I = 0; I <5; I ++); do

{

Echo "$! Start 'date '.";

Sleep 5;

Echo 1> AA & Echo "$! Done 'date '.";

}&

Done

Wait

Cat AA | WC-l

Rm AA

Note:Wait is executed only after all the previous background tasks are completed. Otherwise, the program itself will not wait, which may cause errors for the commands that depend on the results of the previous tasks. For example, the WC-l command above reports an error: the file AA does not exist.

 

[Root @ xml-ora1 ~] # Sh para. Sh

19261 start Mon mar 18 16:47:51 CST 2013.

19264 start Mon mar 18 16:47:51 CST 2013.

19267 start Mon mar 18 16:47:51 CST 2013.

19272 start Mon mar 18 16:47:51 CST 2013.

19269 start Mon mar 18 16:47:51 CST 2013.

19261 done mon mar 18 16:47:56 CST 2013.

19264 done mon mar 18 16:47:56 CST 2013.

19272 done mon mar 18 16:47:56 CST 2013.

19269 done mon mar 18 16:47:56 CST 2013.

19267 done mon mar 18 16:47:56 CST 2013.

5

 

In the uncontrolled mode, all tasks are executed in the background no matter how many tasks exist. That is to say, in this case, the "process" of each task is executed simultaneously. The following information is displayed in the background process:

[Root @ xml-ora1 ~] # Ps-Ef | grep para | grep-V grep

Root 19260 3092 0 00:00:00 pts/2 Sh para. Sh

Root 19261 19260 0 00:00:00 pts/2 Sh para. Sh

Root 19264 19260 0 00:00:00 pts/2 Sh para. Sh

Root 19267 19260 0 00:00:00 pts/2 Sh para. Sh

Root 19269 19260 0 00:00:00 pts/2 Sh para. Sh

Root 19272 19260 0 00:00:00 pts/2 Sh para. Sh

 

Iii. Fine Control of concurrent execution

The instances mentioned above are all situations where the number of processes is uncontrollable. The following describes how to accurately control the number of concurrent processes.

The script establishes inter-process communication by creating a FIFO (named pipeline), and queues commands by using FIFO (first-in-first-out ), the main process keeps executing the commands in the background, and controls the number of concurrent commands to run in the background.

(1) queue commands in FIFO mode and control the number of concurrency;

(2) execute commands in the background. The parent process uses wait to wait for the completion of all child processes;

(3) Get Failure Information and logs;

(4) capture and process signals;

 

[Root @ xml-ora1 ~] # Cat paractl. Sh

#! /Usr/bin/bash

 

# Define the number of concurrent processes

Parallel = 3

# Define the temporary MPs queue file name ($. represents a special PID variable)

Tmpfile =$ $. FIFO

# Define the full path name of the exported configuration file

Export _cfg = $ home/Ptest. cfg

# Defining Failure Log Files

Failure_flag = $ home/fail. Log

 

# Kill sub-process upon interruption (kill-9 0: Kill the script itself and the derived sub-process)

Function trap_exit

{

Kill-9 0

}

 

# Execute command functions

Exec_cmd ()

{

# Here is the actual command to be executed. Sleep is used as an example in this example.

Echo "'date' seq: $ {SEC} $! Start! "

Sleep $ {1}

Echo "'date' seq: $ {SEC} $! Complete! "

Echo

If [$? -Ne 0]

Then

Echo "cmd run fail! "

Return 1

Fi

}

 

# When the signal is 1, 2, 3, 15, execute the command in the trap, that is, call the trap_exit function, and exit the shell and return the signal 2 -- leave 0 and 1 for later use

Trap 'trap _ exit; exit 2' 1 2 3 15

# Clear the failed Identity file

Rm-F $ {failure_flag}

# Create a placeholder for the number of concurrent processes (create a named pipeline)

Mkfifo $ tmpfile

# Specify the object identifier 4 for the named pipe. <> input and output are respectively, that is, the input and output bound to the pipe are all on the object identifier 4!

Exec 4 <> $ tmpfile

 

Rm-F $ tmpfile

 

# Write a newline of $ parallel number to the FIFO. Each carriage return represents a "concurrent Placeholder" (the difference between {} and () indicates whether the shell will derive the sub-process. Let command is used for variable calculation .)

{

Count = $ parallel

While [$ count-GT 0]

Do

Echo "conitue! $ Count"

Let COUNT = $ count-1

Done

}> & 4

 

# Obtain each task in sequence from the task list seq, and read the sleep time from the cfg file in the home directory, that is, the number of tasks to be executed.

While read Sec

Do read <& 4

# Display the process ID of the child id generated by the execution Task Number

Echo "'date' seq: $ {SEC} PID: $"

# Run the main program command or output error log in the background, and clear the identifier after completion

(Exec_cmd $ {SEC} | echo $ {SEC} >$ {failure_flag}; echo> & 4 )&

Done <$ pai_cfg

# Waiting for the return value of the sub-process result

Wait

Exec 4> &-

# Determine if all concurrent processes are successful after the process ends

If [-F $ {failure_flag}]

Then

Exit 1

Else

Exit 0

Fi

 

Configuration File: it indicates the program sleep time, which is actually the number of tasks to be executed. Here, there are three tasks. 5 indicates that the first task is sleep for 5 seconds, 10 sleep for 10 seconds, and 13 sleep for 13 seconds.

[Root @ xml-ora1 ~] # Cat Ptest. cfg

5

10

13

 

First, change the parallel parameter parallel = 3 defined in the parallel script paractl. Sh to parallel = 1 to check the execution in sequence:

[Root @ xml-ora1 ~] # Sh paractl. Sh

Mon mar 1819:30:50CST 2013
SEQ: 5 20450Start!

Mon mar 1819:30:55CST 2013
SEQ: 5 20450Complete!

 

Mon mar 1819:30:55CST 2013 seq: 10 20454 start!

Mon mar 1819:31:05CST 2013 seq: 10 20454 complete!

 

Mon mar 1819:31:05CST 2013 seq: 13 20460 start!

Mon mar 1819:31:18CST 2013 seq: 13 20460 complete!

 

Change the parallel parameter parallel = 1 in the parallel script paractl. Sh to parallel = 2 to check the execution status:

[Root @ xml-ora1 ~] # Sh paractl. Sh

Mon mar 18 19:42:07 CST 2013 seq:5 20578 start!

Mon mar 18 19:42:07 CST 2013 seq: 10 20581 start!

Mon mar 18 19:42:12 CST 2013 seq:5 20578 complete!

 

Mon mar 18 19:42:12 CST 2013 seq:13 20585 start!

Mon mar 18 19:42:17 CST 2013 seq: 10 20581 complete!

 

Mon mar 18 19:42:25 CST 2013 seq: 13 20585 complete!

 

Clearly shows the execution status when the number of concurrent tasks is 1. First, execute the first two tasks. 5 seconds later, after Task 5 is completed, start the task 13, the parallel execution of the three tasks after parallel = 3 is changed to the following:

 

[Root @ xml-ora1 ~] # Sh paractl. Sh

Mon mar 18 19:33:25 CST 2013 seq: 5 20471 start!

Mon mar 18 19:33:25 CST 2013 seq: 10 20474 start!

Mon mar 18 19:33:25 CST 2013 seq: 13 20477 start!

Mon mar 18 19:33:30 CST 2013 seq: 5 20471 complete!

Mon mar 18 19:33:35 CST 2013 seq: 10 20474 complete!

Mon mar 18 19:33:38 CST 2013 seq: 13 20477 complete!

 

Obviously, three tasks are executed simultaneously.

Iv. Concurrent applications

Why is it necessary to study shell concurrency in Linux? I want to test the performance of concurrent processing on my mail server. Will there be performance problems, previously, I helped my colleagues build Sendmail and develop the mliter for forwarding control. The function is complete. However, after all, I am still a DBA instead of dedicated Linux C development, it only took two weeks to change the mliter sample code. Stability over everything.

1. Automatic email sending script

[Root @ xml-ora1 ~] # Cat Telnet. Sh

#! /Usr/bin/CT

Set SMTP [lindex $ argv 0]

Set from [lindex $ argv 1]

Set to [lindex $ argv 2]

Set title [lindex $ argv 3]

Set content [lindex $ argv 4]

Spawn Telnet $ SMTP 25

Secondary CT "220"

Send "HELO xinhua.org \ r"

CT "250 OK"

Send "mail from: <$ from> \ r"

CT "250 mail OK"

Send "rcpt to: <$ to> \ r"

CT "250 mail OK"

Send "data \ r"

Secondary CT "354"

Send "to: $ to \ r"

Send "from: $ from \ r"

Send "Subject: $ title \ r"

Send "\ r"

Send "$ content \ r"

Send ". \ r"

Secondary CT "250"

Send "quit"

 

2. Concurrent email sending script

[Root @ xml-ora1 ~] # Cat parasend. Sh

#! /Bin/sh

For (I = 0; I <80; I ++); do

{

Echo "$ I send start 'date' PID: $ !. ";

 

Echo './telnet. Sh 172.16.28.20 $ I @126.com java3344520@hotmail.com subjects $ I Content'

 

Echo './telnet. Sh 172.16.28.20 410566413@qq.com subjects $ I content $ I'

 

Echo './telnet. Sh 172.16.28.20 $ I @126.com java3344520@hotmail.com subjects $ I Content'

 

Echo './telnet. Sh 172.16.28.20 $ I @qq.com a@xxx.com subjects $ I content $ I'

 

Echo 1> AA & Echo "$ isend complete 'date' PID: $ !. ";

}&

Done

Wait

Cat AA | WC-l

Rm AA

3. Concurrent sending of mail results

Single forwarding-execute scripts in batches

Concurrency on a single server:

Concurrency	Email Processing
50	Normal
80	Normal
100	Failed

 

Because our environment is forwarded by a single server, the number of concurrency is about 80, while the mail server is normally sent by the queue, and the mail server Domino needs to control the number of concurrent sending.

Multiple forwarding methods-execute scripts in batches

In order to test, the script is executed on two client machines and on two clients. If the script runs for 50 times, a memory problem occurs and the filtering condition is exited, the concurrency is stable in 30 cycles, that is, the number of concurrent mails is about 50 per second. Later, we need to improve procedures and improve efficiency.

When the number of concurrent connections on multiple servers is:

Concurrency	Email Processing
20	Normal
30	Normal
50	Failed

 

4. Conclusion

You need to obtain the performance of the forwarding server filter plug-in based on the mail server situation. You need to launch the plug-in for actual testing.

The programming level of the layman can satisfy the needs!