Python Process pool: multiprocessing.pool

In the use of Python for system management, especially the simultaneous operation of multiple file directories, or remote control of multiple hosts, parallel operation can save a lot of time. When the number of objects is small, can be directly used in multiprocessing process dynamic genetic multiple processes, more than 10 is OK, but if it is hundreds, thousands of goals, manual to limit the number of processes is too cumbersome, at this time can play the role of process pool.
Pool can provide a specified number of processes for the user to invoke, and when a new request is submitted to the pool, a new process is created to execute the request if it is not full, but if the number of processes in the pool has reached the specified maximum, the request waits until the process ends in the pool. To create a new process to it.

Example 1: Using a process pool

#Coding:utf-8ImportMultiprocessingImport Timedeffunc (msg):Print "msg:", msg time.sleep (3)    Print "End"if __name__=="__main__": Pool= multiprocessing. Pool (processes = 3)     forIinchXrange (4): Msg="Hello%d"%(i) Pool.apply_async (func, (msg,))#The total number of processes maintained is processes, and a new process is added when a process finishes executing    Print "mark~ mark~ mark~~~~~~~~~~~~~~~~~~~~~~"pool.close () pool.join ( )#call the close function before calling join, otherwise an error will occur. No new processes are added to the Pool,join function to wait for all child processes to end after the close is done    Print "sub-process (es) done."

One execution result

mmsg:hark~ mark~ Mark~~~~~~~~~~~~~~~~~~~~~~ello 0msg:hello 1msg:hello 2endmsg:hello 3endendendSub-process (es) done.

function Explanation :

• Apply_async (func[, args[, kwds[, callback]]) It is non-blocking , apply (func[, args[, Kwds]) is blocked (Understanding the difference, See Example 1 cases 2 result difference)
• Close () Closes the pool so that it is not accepting a new task.
• Terminate () ends the worker process and is not processing the unfinished task.
• Join () The main process is blocked, waiting for the child process to exit, and the Join method is used after close or terminate.

Execution Note : Create a process pool and set the number of processes to 3,xrange (4) will produce four objects in succession [0, 1, 2, 4], four objects are submitted to the pool, because the pool specifies the number of processes is 3, so 0, 1, 2 will be sent directly to the process to execute, The output "Msg:hello 3" appears after "end" when one execution is finished before a Process object 3 is empty. Because it is non-blocking, the main function will do its own self, do not respond to the execution of the process, so run the for loop after the direct output of "mmsg:hark~ mark~ mark~~~~~~~~~~~~~~~~~~~~~~", The main program waits for the end of each process at Pool.join ().

Example 2: Using a process pool (blocking)

#Coding:utf-8ImportMultiprocessingImport Timedeffunc (msg):Print "msg:", msg time.sleep (3)    Print "End"if __name__=="__main__": Pool= multiprocessing. Pool (processes = 3)     forIinchXrange (4): Msg="Hello%d"%(i) Pool.apply (func, (msg,))#The total number of processes maintained is processes, and a new process is added when a process finishes executing    Print "mark~ mark~ mark~~~~~~~~~~~~~~~~~~~~~~"pool.close () pool.join ( )#call the close function before calling join, otherwise an error will occur. No new processes are added to the Pool,join function to wait for all child processes to end after the close is done    Print "sub-process (es) done."

Results of one execution

Msg:hello 0endmsg:hello 1endmsg:hello 2endmsg:hello 3endmark~ mark~ mark~~~~~~~~~~~~~~~~~~~~~~sub-process (es) done.

　　

Example 3: Using a process pool and focusing on the results

ImportMultiprocessingImport Timedeffunc (msg):Print "msg:", msg time.sleep (3)    Print "End"    return " Done"+msgif __name__=="__main__": Pool= multiprocessing. Pool (processes=4) Result= []     forIinchXrange (3): Msg="Hello%d"%(i) Result.append (Pool.apply_async (func, (msg,))) Pool.close () Pool.join () forResinchResult:Print ":::", Res.get ()Print "sub-process (es) done."

One execution result

Msg:hello 0msg:hello 1msg:hello 2endendend::: Donehello 0::: Donehello 1::: Donehello 2sub-process (es) done.

Example 4: Using multiple process pools

#Coding:utf-8ImportMultiprocessingImportOS, time, RandomdefLee ():Print "\nrun Task lee-%s"% (Os.getpid ())#Os.getpid () Gets the ID of the current processStart =time.time () time.sleep (Random.random ()* 10)#random.random () randomly generates a decimal between 0-1End =time.time ()Print 'Task Lee, runs%0.2f seconds.'% (End-start)defMarlon ():Print "\nrun Task marlon-%s"%(Os.getpid ()) Start=time.time () time.sleep (Random.random ()* 40) End=time.time ()Print 'Task Marlon runs%0.2f seconds.'% (End-start)defAllen ():Print "\nrun Task allen-%s"%(Os.getpid ()) Start=time.time () time.sleep (Random.random ()* 30) End=time.time ()Print 'Task Allen runs%0.2f seconds.'% (End-start)defFrank ():Print "\nrun Task frank-%s"%(Os.getpid ()) Start=time.time () time.sleep (Random.random ()* 20) End=time.time ()Print 'Task Frank runs%0.2f seconds.'% (End-start)if __name__=='__main__': Function_list=[Lee, Marlon, Allen, Frank]Print "Parent Process%s"%(Os.getpid ()) Pool=multiprocessing. Pool (4)     forFuncinchFunction_list:pool.apply_async (func)#pool execution function, apply execution function, when a process is completed, a new process is added to the pool    Print 'waiting-subprocesses done ...'pool.close () pool.join ( )#before calling join, be sure to call the close () function, otherwise there will be an error, and close () will not have a new process joined to the Pool,join function waiting for the end of the child process .    Print 'All subprocesses is done .'

One execution result

Parent process 7704Waiting for all subprocesses-done ... The Run Task Lee-6948run Task marlon-2896run Task Allen-7304run Task Frank-3052task Lee, runs 1.59 seconds. Task Marlon runs 8.48 seconds. Task Frank runs 15.68 seconds. Task Allen runs 18.08 seconds. All subprocesses is done.

Python Process pool: multiprocessing.pool