Python process pool: multiprocessing. pool,

Python process pool: multiprocessing. pool,

This article goes to workshop.

When using Python for system management, especially operating multiple file directories at the same time, or remotely controlling multiple hosts, parallel operations can save a lot of time. When the number of objects to be operated is small, you can use the Process in multiprocessing to dynamically generate multiple processes. A dozen processes are fine, but if there are hundreds or thousands of targets, manually limiting the number of processes is too cumbersome. In this case, you can use the process pool function.
The Pool can provide a specified number of processes for users to call. When a new request is submitted to the pool, if the Pool is not full, a new process is created to execute the request; however, if the number of processes in the pool has reached the specified maximum value, the request will wait until a process in the pool ends and a new process will be created.

Example 1: Process poolFrom multiprocessing import freeze_support, Poolimport timedef Foo (I): time. sleep (2) print ('___ time ---', time. ctime () return I + 100def Bar (arg): print ('---- exec done:', arg, time. ctime () if _ name _ = '_ main _': freeze_support () pool = Pool (3) # The number of concurrent processes in the thread pool is 3 for I in range (4): pool. apply_async (func = Foo, args = (I,), callback = Bar) # The number of concurrent processes in the thread pool is 3. After a process is executed, if a new process is waiting for execution, it will be added to # pool. apply (func = Foo, ar Gs = (I,) print ('end') pool. close () pool. join () # Call the close function before calling join. Otherwise, an error occurs. After close is executed, no new processes will be added to the pool. The join function waits for the completion of all sub-processes.View Code

Execution result:

end___time--- Thu Jun 16 15:11:45 2016----exec done: 100 Thu Jun 16 15:11:45 2016___time--- Thu Jun 16 15:11:45 2016----exec done: 101 Thu Jun 16 15:11:45 2016___time--- Thu Jun 16 15:11:45 2016----exec done: 102 Thu Jun 16 15:11:45 2016___time--- Thu Jun 16 15:11:47 2016----exec done: 103 Thu Jun 16 15:11:47 2016

Function explanation:

• Apply_async (func [, args [, kwds [, callback]) it isNon-blocking, Apply (func [, args [, kwds]) isBlocking(To understand the difference, see Example 1 and Example 2)
• Close () closes the pool so that it does not accept new tasks.
• Terminate () ends the working process and does not process unfinished tasks.
• The join () main process is blocked and waits for the sub-process to exit. The join method should be used after close or terminate.

Execution instructions: Create a process pool and set the number of processes to 3. xrange (4) generates four objects [0, 1, 2, 4] one after another. four objects are submitted to the pool. because the number of processes specified by the pool is 3, 0, 1, and 2 are directly sent to the process for execution, when one of the processes is finished, a process processing object 3 is blank, so the output "msg: hello 3" appears after "end. Because it is non-blocking, the main function will execute its own and ignore the execution of the process. Therefore, after running the for loop, the "mMsg: hark ~ Mark ~ Mark ~~~~~~~~~~~~~~~~~~~~~~", The main program waits for the completion of each process at pool. join.

Example 2: Process pool (blocking)From multiprocessing import freeze_support, Poolimport timedef Foo (I): time. sleep (2) print ('___ time ---', time. ctime () return I + 100def Bar (arg): print ('---- exec done:', arg, time. ctime () if _ name _ = '_ main _': freeze_support () pool = Pool (3) # The number of concurrent processes in the thread pool is 3 for I in range (4): pool. apply (func = Foo, args = (I,) print ('end') pool. close () pool. join () # Call the close function before calling join. Otherwise, an error occurs. After close is executed, no new processes will be added to the pool. The join function waits for the completion of all sub-processes.View Code

Execution result

___time--- Thu Jun 16 15:15:16 2016___time--- Thu Jun 16 15:15:18 2016___time--- Thu Jun 16 15:15:20 2016___time--- Thu Jun 16 15:15:22 2016end

Example 3: Use the process pool and follow the resultsImport multiprocessingimport timedef func (msg): print ('Hello: ', msg, time. ctime () time. sleep (2) print ('end', time. ctime () return 'done' + msgif _ name __= = '_ main _': pool = multiprocessing. pool (2) result = [] for I in range (3): msg = 'Hello % s' % I result. append (pool. apply_async (func = func, args = (msg,) pool. close () pool. join () for res in result: print ('***:', res. get () print ('aaaaaall end --')View Code

Execution result

Hello: hello 0 Thu Jun 16 15:26:33 2016
Hello: hello 1 Thu Jun 16 15:26:33 2016
End Thu Jun 16 15:26:35 2016
Hello: hello 2 Thu Jun 16 15:26:35 2016
End Thu Jun 16 15:26:35 2016
End Thu Jun 16 15:26:37 2016
* **: Donehello 0
* **: Donehello 1
* **: Donehello 2
AAAAAAAAll end --

 Note: The get () function returns the value of each returned result.

Example 4: Use multiple process poolsImport multiprocessingimport time, OS, randomdef Lee (): print ('\ nRun task Lee -- % s ****** ppid: % s' % (OS. getpid (), OS. getppid ()),'~~~~ ', Time. ctime () start = time. time () time. sleep (random. randrange (10) end = time. time () print ('Task Lee, runs % 0.2f seconds. '% (end-start ),'~~~~ ', Time. ctime () def Marlon (): print ("\ nRun task Marlon-% s ******* ppid: % s" % (OS. getpid (), OS. getppid ()),'~~~~ ', Time. ctime () start = time. time () time. sleep (random. random () * 40) end = time. time () print ('Task Marlon runs % 0.2f seconds. '% (end-start ),'~~~~ ', Time. ctime () def Allen (): print ("\ nRun task Allen-% s ******* ppid: % s" % (OS. getpid (), OS. getppid ()),'~~~~ ', Time. ctime () start = time. time () time. sleep (random. random () * 30) end = time. time () print ('Task Allen runs % 0.2f seconds. '% (end-start ),'~~~~ ', Time. ctime () def Frank (): print ("\ nRun task Frank-% s ******* ppid: % s" % (OS. getpid (), OS. getppid ()),'~~~~ ', Time. ctime () start = time. time () time. sleep (random. random () * 20) end = time. time () print ('Task Frank runs % 0.2f seconds. '% (end-start ),'~~~~ ', Time. ctime () if _ name _ = '_ main _': func_list = [Lee, Marlon, Allen, frank] print ('parent process id % s' % OS. getpid () pool = multiprocessing. pool (4) for func in func_list: pool. apply_async (func) # Pool executes the function and applies executes the function. After a process is executed, will add a new process to the pool print ('Waiting for all subprocesses done... ') pool. close () pool. join () # Before calling join, you must call the close () function first. Otherwise, an error occurs. No new process is added to the pool after close () is executed, the join function waits for print ('all subprocesses done. ')View Code

Execution result

parent process id 98552Waiting for all subprocesses done...Run task Lee--97316******ppid:98552 ~~~~ Thu Jun 16 15:20:50 2016Run task Marlon-95536******ppid:98552 ~~~~ Thu Jun 16 15:20:50 2016Run task Allen-95720******ppid:98552 ~~~~ Thu Jun 16 15:20:50 2016Run task Frank-98784******ppid:98552 ~~~~ Thu Jun 16 15:20:50 2016Task Allen runs 0.31 seconds. ~~~~ Thu Jun 16 15:20:51 2016Task Lee,runs 7.00 seconds. ~~~~ Thu Jun 16 15:20:57 2016Task Frank runs 14.48 seconds. ~~~~ Thu Jun 16 15:21:05 2016Task Marlon runs 31.72 seconds. ~~~~ Thu Jun 16 15:21:22 2016All subprocesses done.

Multiprocessing pool map

#coding: utf-8import multiprocessing def m1(x):     print x * x if __name__ == '__main__':     pool = multiprocessing.Pool(multiprocessing.cpu_count())     i_list = range(8)    pool.map(m1, i_list)

Execution result

0
1
4
9
16
25
36
49

Reference: http://www.dotblogs.com.tw/rickyteng/archive/2012/02/20/69635.aspx

 

Problem: http://bbs.chinaunix.net/thread-4111379-1-1.html

#coding: utf-8import multiprocessingimport loggingdef create_logger(i):    print iclass CreateLogger(object):    def __init__(self, func):        self.func = funcif __name__ == '__main__':    ilist = range(10)    cl = CreateLogger(create_logger)    pool = multiprocessing.Pool(multiprocessing.cpu_count())    pool.map(cl.func, ilist)    print "hello------------>"

Execution result

0
1
2
3
4
5
6
7
8
9
Hello ------------>