Application of process pooling pool in Python multi-process concurrency operation

In the use of Python for system management, especially the simultaneous operation of multiple file directories, or remote control of multiple hosts, parallel operation can save a lot of time. When the number of operands is small, you can directly use the process of multiprocessing in the dynamic genetic multiple processes, 10 is OK, but if it is hundreds, thousands of goals, manual to limit the number of processes is too cumbersome, It's time for the process pool to work.
The pool can provide a specified number of processes for the user to call, and when a new request is submitted to the pools, if it is not full,

A new process is created to execute the request, but if the number of processes in the pool has reached the specified maximum,

Then the request waits until the process ends in the pool,

To create a new process to it. Here's a simple example:

#!/usr/bin/env python#coding=utf-8from multiprocessing Import poolfrom time import sleepdef f (x): for    I in range (10):        print '%s---%s '% (i, x)        sleep (1) def main ():    pool = Pool (processes=3)    # Set the processes max number 3 For    i in range (11,20):        result = Pool.apply_async (f, (I,))    pool.close ()    pool.join ()    if Result.successful ():        print ' successful ' if __name__ = = "__main__":    Main ()

Create a process pool with a capacity of 3 , and then pass F (i) to it in turn, using PS aux after running the script | grep pool.py looks at the process and finds that only three processes are executing. Pool.apply_async () is used to submit a target request to the process pool,pool.join () is used to wait for the worker process in the process pool to complete, preventing the main process from ending before the worker process ends. However, the Pool.join () must be used after pool.close () or Pool.terminate ( ). the difference between close () and terminate () is that close () waits for the worker process in the pool to end and then close the pool, while terminate () is closed directly. Result.successful () indicates the state of the entire invocation execution, and throws a Assertionerror exception if there are still workers that have not finished executing .
With the pool of multiprocessing, it is easy to automate hundreds of or thousands of parallel operations at the same time, and the complexity of scripting is greatly reduced.

Introduction to Multiprocessing.pool functions in Python

One apply(func[, args[, Kwds])
apply is used to pass an indeterminate parameter, consistent with the Apply function in Python (although the built-in apply function is not recommended since 2.3), the main process blocks the function.
For x in Gen_list (l):
result = Pool.apply (Pool_test, (x,))
print ' main process '
at this point, the execution process of the main process is consistent with the single process
two apply_async(func[, args[, kwds[, callback]])
is consistent with the apply usage, but it is non-blocking and supports callbacks after the result is returned.
For x in Gen_list (l):
result = Pool.apply_async (Pool_test, (x,))
print ' main process '
this time the main process loop runs without waiting for the return result of Apply_async, and exits after the main process is finished, even if the child process has not returned the entire program. Although Apply_async is non-blocking, the Get method that returns the result is blocked, and in this case result.get () blocks the main process. So you can handle the return result as follows:
[X.get () for x in [Pool.apply_async (Pool_test, (x,))-X in Gen_list (l)]]
If we are not interested in returning results, we can use Pool.close and Pool.join in the main process to prevent the main process from exiting. Note the Join method must be called after close or terminate.
For x in Gen_list (l):
Pool.apply_async (Pool_test, (x,))
print ' main_process '
pool.close ()
Pool.join ()
three map(func, iterable[, chunksize])
The map method is basically consistent with the built-in map function behavior, which causes the process to block with this until the result is returned.
However, it is important to note that although the second parameter is described as iterable, in practice it is found that the program will run the child process only after the entire queue is fully ready.
four Map_async(func, iterable[, chunksize[, callback])
consistent with the map usage, but it is non-blocking. See Apply_async for its related matters.
FIVE IMAP(func, iterable[, chunksize])
Unlike map, the return result of IMAP is ITER, which requires the active use of next in the main process to drive child process calls. Even if the child process does not return a result, the main process will continue for gen_list (L) ITER, and according to the python2.6 document description, the Chunksize setting will be larger than the default of 1 for large data volumes iterable.
for X in Pool.imap (Pool_test, Gen_list (L)):
Pass
six imap_unordered(func, iterable[, chunksize])
consistent with IMAP, except that it does not guarantee that the return result is consistent with the order in which the iteration is passed in.
seven Close ()
Close the pool so that it is not accepting new tasks.
eight terminate ()
ends a worker process and is not processing an unhandled task.
Nine join ()
The main process blocks waiting for the child process to exit, and the Join method is used after close or terminate.

L = Range (10)
def gen_list (L):
For X in L:
print ' yield ', X
Yield x

def pool_test (x):
print ' F2 ', X
Time.sleep (1)

Application of process pooling pool in Python multi-process concurrency operation