Python Process Pool

First, the definition:

Multi-process is one of the means to achieve concurrency, in the use of Python system management, especially at the same time to operate multiple file directories, or remote control of multiple hosts, parallel operation can save a lot of time. Pool ([numprocess [, initializer [, Initargs]]): Create a process pool

Second, the main parameters:

1 numprocess: The number of processes to be created, and if omitted, the value of Cpu_count () will be used by default of 2 initializer: The callable object to execute when each worker process starts, default to None 3 Initargs: is the parameter group to pass to the initializer

Third, the main methods:

P.apply (func [, args [, Kwargs]): Executes func (*args,** Kwargs) in a pool worker processand returns the result. It should be emphasized that this operation does not execute the Func function in all pool worker processes. If you want to execute the Func function concurrently with different parameters, you must call the P.apply () function from a different thread or use the P.apply_async () P.apply_async (func [, args [, Kwargs]]): Executes func (*args,** Kwargs) in a pool worker processand returns the result. The result of this method is an instance of the AsyncResult class, and callback is a callable object that receives input parameters. When the result of Func becomes available, the understanding is passed to callback. Callback does not prohibit any blocking operations, otherwise it will receive results from other asynchronous operations.   P.close (): Closes the process pool to prevent further action. If all operations persist, they will complete p.jion () before the worker process terminates: waits for all worker processes to exit. This method can only be called after close () or teminate ()

Iv. Other methods:

The return value of Method Apply_async () and Map_async () is an instance of Asyncresul obj. The instance has the following method Obj.get (): Returns the result and waits for the result to arrive if necessary. Timeout is optional. If it has not arrived within the specified time, a one will be raised. If an exception is thrown in a remote operation, it is raised again when this method is called. Obj.ready (): If the call is complete, return trueobj.successful (): Returns True if the call completes without throwing an exception, or if this method is called before the result is ready, throws an exception obj.wait ([timeout]): Waits for the result to become available. Obj.terminate (): Immediately terminates all worker processes without performing any cleanup or end of any pending work. If P is garbage collected, this function is called automatically

V. Examples:

 fromMultiprocessingImportPoolImportOs,timedefWork (n):Print('task <%s> is runing'%os.getpid ()) Time.sleep (2)    returnN**2if __name__=='__main__':    #print (Os.cpu_count ())P=pool (4)#execute several processes in parallel    #For I in Range (Ten):    #res=p.apply (work,args= (i,))    #Print (res)res_l=[]     forIinchRange (10): Res=p.apply_async (work,args=(i,)) Res_l.append (res) p.close () P.join ()#    #For Res in res_l:    #print (Res.get ())

Six, callback function:

ImportRequests#PIP3 Install requestsImportOs,time fromMultiprocessingImportPooldefget_page (URL):Print('<%s> get:%s'%(Os.getpid (), URL)) Respone=requests.get (URL)ifRespone.status_code = = 200:        return{'URL': URL,'text': Respone.text}defParse_page (DIC):Print('<%s> Parse:%s'% (Os.getpid (), dic['URL'])) Time.sleep (0.5) Res='url:%s size:%s\n'% (dic['URL'],len (dic['text']))#simulate parsing web contentWith open ('Db.txt','a') as F:f.write (res)if __name__=='__main__': P=pool (4) URLs= [        'http://www.baidu.com',        'http://www.baidu.com',        'http://www.baidu.com',        'http://www.baidu.com',        'http://www.baidu.com',        'http://www.baidu.com',        'http://www.baidu.com',    ]     forUrlinchUrls:p.apply_async (Get_page,args= (URL,), callback=parse_page)#Callback the resulting results to the return seasoning function for processingp.close () p.join ( )Print('main process PID:', Os.getpid ())

Vii. concurrent communication instances of Process pool control set bytes

#Client fromSocketImport*C=socket (Af_inet,sock_stream) c.connect (('127.0.0.1', 8080)) whiletrue:msg=input ('>>:'). Strip ()if  notMsgContinuec.send (Msg.encode ('Utf-8')) Data=C.RECV (1024)    Print(Data.decode ('Utf-8') ) C.close ()

#服务端
fromMultiprocessingImportPoolImportOS fromSocketImport*s=socket (af_inet,sock_stream) s.setsockopt (sol_socket,so_reuseaddr,1) S.bind ('127.0.0.1', 8080)) S.listen (5)defTalk (conn,addr):Print(Os.getpid ()) whileTrue:Try: Data=CONN.RECV (1024)            if  notData BreakConn.send (Data.upper ())exceptException: Breakconn.close ()if __name__=='__main__': P=pool (4)     whiletrue:conn,addr=s.accept () p.apply_async (Talk,args=(CONN,ADDR)) S.close ()

Python Process Pool