Using Concurrent.futures module concurrency, implement process pool, thread pool

First, about the Concurrent.futures module

The Python standard library provides us with the threading and multiprocessing modules to write the corresponding asynchronous multithreaded/multi-process code. Starting with Python3.2, the standard library provides us with concurrent.futures modules that provide ThreadPoolExecutor and ProcessPoolExecutor two classes of Threadpoolexecutor and processpoolexecutor inherit executor, Code that is used to create the thread pool and process pool, respectively. A more advanced abstraction of the pair is implemented threading multiprocessing , providing direct support for writing thread pool/process pools.
Concurrent.futures base modules are executor and future.

The base of the Concurrent.futures module is exectuor, executor is an abstract class that cannot be used directly. However, the two subclass Threadpoolexecutor and processpoolexecutor that it provides are useful, as the name implies, to create the code for the thread pool and process pool, respectively. We can put the corresponding tasks directly into the thread pool/process pool, do not need to maintain queue to worry about deadlock problem, the thread pool/process pool will automatically help us dispatch.

Future This concept is believed to have the Java and Nodejs programming experience the friend certainly is not unfamiliar, you can interpret it as a work that completes in the next , this is the Asynchronous Programming Foundation, In the traditional programming mode, for example, when we operate the queue.get, there will be blocking before we wait for the results to be returned, and the CPU cannot let out other things, and the introduction of the future helps us to complete other operations during the waiting period.

The method is defined in executor submit() , which is the function of committing an executable callback task and returning a future instance . The future object represents the given invocation.

Two submit()method to implement the process pool/thread pool process pools

 fromConcurrent.futuresImportProcesspoolexecutorImportOs,time,randomdeftask (n):Print('%s is running'%os.getpid ()) Time.sleep (2)    returnN**2if __name__=='__main__': P=processpoolexecutor () #不填则默认为cpu的个数 l=[] Start=time.time () forIinchRange (10): obj=p.submit (Task,i)#the Submit () method returns a future instance that needs to be obj.result () to get the resultl.append (obj) p.shutdown ()#similar to the role of close and join in the process pool implemented with the From multiprocessing import pool    Print('='*30)    #print ([obj for obj in L])    Print([Obj.result () forObjinchl])Print(Time.time ()-start)#The above method can also be written as follows    #start = Time.time ()    #with Processpoolexecutor () as P: #类似打开文件, can be omitted. Shutdown ()    #future_tasks = [P.submit (task, I) for I in Range (Ten)]    #print (' = ' *)    #print ([Obj.result () for obj in Future_tasks])    #print (Time.time ()-start)

Thread pool

 fromConcurrent.futuresImportProcesspoolexecutor,threadpoolexecutorImportThreadingImportOs,time,randomdeftask (n):Print('%s:%s is running'%(Threading.currentthread (). GetName (), Os.getpid ())) Time.sleep (2)    returnN**2if __name__=='__main__': P=threadpoolexecutor () #不填则默认为cpu的个数 l=[] Start=time.time () forIinchRange (10): obj=p.submit (task,i) l.append (obj) p.shutdown ()Print('='*30)    Print([Obj.result () forObjinchl])Print(Time.time ()-start)#The above method can also be written as follows    #start = Time.time ()    #with Threadpoolexecutor () as P: #类似打开文件, can be omitted. Shutdown ()    #future_tasks = [P.submit (task, I) for I in Range (Ten)]    #print (' = ' *)    #print ([Obj.result () for obj in Future_tasks])    #print (Time.time ()-start)

The default is asynchronous execution

#P.submit (task,i). Result () is synchronous execution fromConcurrent.futuresImportProcesspoolexecutor,threadpoolexecutorImportOs,time,randomdeftask (n):Print('%s is running'%os.getpid ()) Time.sleep (2)    returnN**2if __name__=='__main__': P=processpoolexecutor () Start=time.time () forIinchRange (10): Res=P.submit (task,i). Result ()Print(RES)Print('='*30)    Print(Time.time ()-start)

Third, callback function

 fromConcurrent.futuresImportProcesspoolexecutor,threadpoolexecutorImportRequestsImportOSImport Time fromThreadingImportCurrentThreaddefget_page (URL):Print('%s:<%s> is getting [%s]'%(CurrentThread (). GetName (), Os.getpid (), URL)) Response=requests.get (URL) time.sleep (2)    return{'URL': URL,'text': Response.text}defParse_page (RES):#The res Here is a future object obtained by a p.submit, not a resultRes=res.result ()#Res.result () got the corresponding results.    Print('%s:<%s> Parse [%s]'% (CurrentThread (). GetName (), Os.getpid (), res['URL'])) with open ('Db.txt','a') as F:parse_res='url:%s size:%s\n'% (res['URL'],len (res['text'])) F.write (parse_res)if __name__=='__main__':    #P=processpoolexecutor ()p=threadpoolexecutor () URLs= [        'https://www.baidu.com',        'https://www.baidu.com',        'https://www.baidu.com',        'https://www.baidu.com',        'https://www.baidu.com',        'https://www.baidu.com',    ]     forUrlinchURLs:#multiprocessing.pool_obj.apply_async (get_page,args= (URL,), callback=parse_page)P.submit (get_page, URL). Add_done_callback (Parse_page)#Unlike the previous callback function, the result is that the object returned after the previous submit method has been executed. Result to get the corresponding resultP.shutdown ()Print('Master', Os.getpid ())

Four, map method

Similar to the built-in function map usage, this method returns a map (func, *iterables) iterator, and the callback in the iterator executes the returned result in an orderly manner.

The following is the implementation of the process pool, thread pool through the map method of the objects instantiated by class Threadpoolexecutor and Processpoolexecutor under the Concurrent.futures module

 fromConcurrent.futuresImportProcesspoolexecutor,threadpoolexecutorImportOs,timedeftask (n):Print('%s is running'%os.getpid ()) Time.sleep (2)    returnN**2if __name__=='__main__':    #P=processpoolexecutor ()p=threadpoolexecutor () Start=time.time () obj=p.map (Task,range (10) ) P.shutdown ()Print('='*30)    Print(list (obj))Print(Time.time ()-start)

Using Concurrent.futures module concurrency, implement process pool, thread pool