Python concurrent programming thread pool/process Pool--concurrent.futures module

First, about the Concurrent.futures module

　　

　　The Python standard library provides us with the threading and multiprocessing modules to write the corresponding multithreaded/multi-process code, but when the project reaches a certain scale, frequent creation/destruction of processes or threads is very resource intensive, and this time we will write our own thread pool/ Process pool, with space to change time. But starting with Python3.2, the standard library provides us with the concurrent.futures module, which providestwo classes of ThreadPool executor and Processpoolexecutor, Further abstraction of threading and multiprocessing is achieved, providing direct support for writing thread pool/process pools.

1.Executor and Future:

The base of the Concurrent.futures module is exectuor, executor is an abstract class that cannot be used directly. However, the two subclass Threadpoolexecutor and processpoolexecutor that it provides are useful, as the name implies, to create the code for the thread pool and process pool, respectively. We can put the corresponding tasks directly into the thread pool/process pool, do not need to maintain queue to worry about deadlock problem, the thread pool/process pool will automatically help us dispatch.

Future This concept is believed to have the Java and Nodejs programming experience the friend certainly is not unfamiliar, you can interpret it as a work that completes in the next , this is the Asynchronous Programming Foundation, In the traditional programming mode, for example, when we operate the queue.get, there will be blocking before we wait for the results to be returned, and the CPU cannot let out other things, and the introduction of the future helps us to complete other operations during the waiting period.

P.S: If you are still sticking to python2.x, install the Futures module first.

Second, the operation thread pool/process Pool1. Use submit to operate the thread pool/process pool:

#thread pool: fromConcurrent.futuresImportThreadpoolexecutorImportUrllib.requesturls= ['http://www.163.com','https://www.baidu.com/','https://github.com/']defload_url (URL): With Urllib.request.urlopen (URL, timeout=60) as Conn:Print('%r page is%d bytes'%(URL, len (Conn.read () ))) executor= Threadpoolexecutor (max_workers=3) forUrlinchurls:future=executor.submit (Load_url,url)Print(Future.done ())Print('Main Thread')#Operation Result:falsefalsefalse main thread'https://www.baidu.com/'Page is227bytes'http://www.163.com'Page is662047bytes'https://github.com/'Page is54629 bytes

We analyze it according to the results of the operation. We use the Submit method to add a task,submit to the thread pool to return a future object, which can be simply understood as an operation to be done in the next. because the thread pool commits the task asynchronously, the main thread does not wait for the threads constructor to finish executing, so print (' main thread ') is executed, and the thread created in the corresponding thread pool is not finished, so Future.done () returns the result to False.

#Process Pool: Ibid . fromConcurrent.futuresImportProcesspoolexecutorImportUrllib.requesturls= ['http://www.163.com','https://www.baidu.com/','https://github.com/']defload_url (URL): With Urllib.request.urlopen (URL, timeout=60) as Conn:Print('%r page is%d bytes'%(URL, len (Conn.read () ))) executor= Processpoolexecutor (max_workers=3)if __name__=='__main__': # to add main      forwr.inchurls:future=executor.submit (Load_url,url)Print(Future.done ())Print('Main Thread')#Operation Result:False#child processes are created only and do not completefalse False The main thread # child process creation completes the main thread down and does not wait for the child process to finish executing'http://www.163.com'Page is662049bytes'https://www.baidu.com/'Page is227bytes'https://github.com/'Page is54629 bytes

2. use map to manipulate thread pool/process pools:

In addition to Submit,exectuor, we are provided with a map method similar to the built-in map usage:

 fromConcurrent.futuresImportThreadpoolexecutorImportUrllib.requesturls= ['http://www.163.com','https://www.baidu.com/','https://github.com/']defload_url (URL): With Urllib.request.urlopen (URL, timeout=60) as Conn:Print('%r page is%d bytes'%(URL, len (Conn.read () ))) executor= Threadpoolexecutor (max_workers=3) Executor.map (load_url,urls)Print('Main Thread')#Operation Result:Main Thread'http://www.163.com'Page is662047bytes'https://www.baidu.com/'Page is227bytes'https://github.com/'Page is54629 bytes

As you can see from the running results, themap is returned in the order of the URLs list elements , and the code that is written is more concise and intuitive, and we can choose one based on the specific requirements.

3.wait:

The wait method then returns a tuple (tuple) with two sets (set), one completed (completed) and the other uncompleted (unfinished). One of the advantages of using the wait method is to gain greater freedom, which receives three parameters first_completed, First_exception , and All_complete, which are set by default to All_ Completed.

　　If the default all_completed is used, the program blocks until all the tasks in the thread pool are complete, and then the main thread is executed:

 fromConcurrent.futuresImportthreadpoolexecutor,wait,as_completedImportUrllib.requesturls= ['http://www.163.com','https://www.baidu.com/','https://github.com/']defload_url (URL): With Urllib.request.urlopen (URL, timeout=60) as Conn:Print('%r page is%d bytes'%(URL, len (Conn.read () ))) executor= Threadpoolexecutor (max_workers=3) F_list= [] forwr.inchurls:future=executor.submit (Load_url,url) f_list.append (future)Print(Wait (f_list))Print('Main Thread')#Operation Result:'http://www.163.com'Page is662047bytes'https://www.baidu.com/'Page is227bytes'https://github.com/'Page is54629Bytesdoneandnotdonefutures ( done={<future at 0x2d0f898 State=finished returned Nonetype>, <future at 0x2bd0630 State=finished returned NoneType& gt;, <future at 0x2d27470 state=finished returned Nonetype>}, not_done=set ()) main thread

If the first_completed parameter is used, the program does not wait for all tasks in the thread pool to complete.

 fromConcurrent.futuresImportthreadpoolexecutor,wait,as_completedImportUrllib.requesturls= ['http://www.163.com','https://www.baidu.com/','https://github.com/']defload_url (URL): With Urllib.request.urlopen (URL, timeout=60) as Conn:Print('%r page is%d bytes'%(URL, len (Conn.read () ))) executor= Threadpoolexecutor (max_workers=3) F_list= [] forwr.inchurls:future=executor.submit (Load_url,url) f_list.append (future)Print(Wait (f_list,return_when='first_completed'))Print('Main Thread')#Operation Result:'http://www.163.com'Page is662047Bytesdoneandnotdonefutures ( done={<future at 0x2bd15c0 state=finished returned Nonetype>},Not_done={<future at 0x2d0d828 State=running>, <future at 0x2d27358 state=running>}) Main thread'https://www.baidu.com/'Page is227bytes'https://github.com/'Page is54629 bytes

？ Write a small program to compare Multiprocessing.pool (ThreadPool) and Processpollexecutor (Threadpoolexecutor) in the execution efficiency gap, Why would this result be combined with the future thinking mentioned above?

Python concurrent programming thread pool/process Pool--concurrent.futures module