Python Process Pool

Process Pooling: Parallel operations can save a lot of time when using Python for system management, especially when operating multiple file directories at the same time, or remotely controlling multiple hosts.      When the number of objects is small, can be directly used in multiprocessing process dynamic genetic multiple processes, more than 10 is ok, but if it is hundreds, thousands of goals, manual to limit the number of processes is too cumbersome, at this time can play the role of process Pool. Pool can provide a specified number of processes for the user to invoke, and when a new request is submitted to the pool, a new process is created to execute the request if it is not full, but if the number of processes in the pool has reached the specified maximum, the request waits until the process ends in the Pool. To create a new process to it. Pool Correlation function

1. Apply (func[, args[, kwds])
Apply is used to pass an indeterminate parameter, consistent with the Apply function in Python (although the built-in apply function is not recommended since 2.3), the main process blocks the Function.

The execution process of the main process is consistent with the single process.

2, apply_async (func[, args[, kwds[, callback]])
is consistent with the Apply usage, but it is non-blocking and supports callbacks after the result is Returned.

The main process loop runs without waiting for the return result of the apply_async, and exits when the main process finishes, even if the child process has not returned the entire Program. Although Apply_async is non-blocking, the Get method that returns the result is blocked, such as using Result.get () to block the main process.
If we are not interested in returning results, we can use Pool.close and Pool.join in the main process to prevent the main process from Exiting. Note the Join method must be called after close or Terminate.

3. Map (func, iterable[, Chunksize])
The map method is functionally equivalent to the built-in map (), except that a single task runs in Parallel. It blocks the process until the result is Returned.
however, It is important to note that although the second parameter is described as iterable, in practice it is found that the program will run the child process only after the entire queue is fully ready.

4, map_async (func, iterable[, chunksize[, callback])
Consistent with the map usage, but it is non-blocking. See Apply_async for its related matters.

5. IMAP (func, iterable[, Chunksize])
Unlike map, the return result of IMAP is iter, which requires the active use of next in the main process to drive child process Calls. Even if the child process does not return a result, the main process will continue for gen_list (l) iter, and according to the python2.6 document description, the Chunksize setting will be larger than the default of 1 for large data volumes Iterable.
For x in Pool.imap (pool_test, gen_list (l)):
Pass

6, imap_unordered (func, iterable[, chunksize])
Consistent with imap, except that it does not guarantee that the return result is consistent with the order in which the iteration is passed IN.

7. Close ()
Close the pool so that it no longer accepts new tasks.

8, Terminate ()
Ends the worker process and no longer handles the unhandled Task.

9. Join ()
The main process blocks waiting for the child process to exit, and the Join method is used after close or Terminate.

Cases:

# -*- coding: UTF-8-*-

From Gevent.pool Import Pool
From gevent Import Monkey
Monkey.patch_all ()

#要在调用进程池执行的函数

def sayHi(num):

  print "def print result:",num

#进程池最大运行数

p = Pool(4)

#模拟并发调用线程池

fori in range(10):

  p.map(sayHi,[i])


Execution Result:

def print result:0

def print Result:1

def print Result:2

def print Result:3

def print Result:4

def print Result:5

def print Result:6

def print Result:7

def print Result:8

def print Result:9

Python Process Pool