# Process Pool # Why do you have a process pool? #在程序实际处理任务过程中, there are thousands of tasks that need to be performed when busy, and there may be sporadic tasks at leisure. #那么在成千上万个任务需要执行的时候, do we need to create thousands of processes? #首先, creating a process consumes time, The destruction process also consumes time. Second, even if thousands of processes are turned on, the operating system does not allow them to execute simultaneously, which in turn affects the efficiency of the program. So we can't be unlimited depending on the # task or the end of the task. #在这里, to end a process pool concept, define the good one pool, put a fixed number of # in the process, there is a need to come, just in a pool process to handle the task, wait until the task is processed, the process does not close # but put the process back into the pool to continue waiting for the task. If there are many tasks to be performed, The number of processes in the pool is not enough, and the task requires # Wait for the previous process to complete the task to return, to get the idle process to continue execution. This increases the operating system scheduling difficulty # also saves the opening and closing process time, also to some extent can achieve concurrency effect # multiprocessing. Pool Module #pool ([numprocess [, initializer [, Initargs]]): Create process Pool # parameter Introduction #numprocess: Number of processes to be created, if omitted, the value of Cpu_count () will be used by default ( This is a method of the OS module) #initializer: Is the callable object to execute at the start of each worker process, the default is None#initargs: is the parameter group to pass to initializer # There are three main methods: # p = pool#1.p.apply ( func [, args [, Kwargs]]: executes func (*args,**kwargs) in a pool worker process and returns the result # Note: This operation does not execute the Func function in all pool worker processes. If you want to execute the Func function concurrently with different parameters # You must call the P.apply () function from a different thread or use P.apply_async () # Synchronization, and only Func will continue executing the code after it is executed, return value is Func's return value # Synchronous processing task, all processes in the process pool are normal process # 2. P.map (Self,func,iterable,chunksize=none): # async, comes with close and join, return value for Func returns a list of # 3. P.apply_async (func [, args [, Kwargs]): Executes func (*args,**kwargs) in a pool worker process #And then return the result # The result of this method is an instance of the AsyncResult class, callback is a callable object that receives the input parameter, and when the result of Func becomes available it is # to pass the result to callback. Callback prohibit any blocking operation, no one will receive other asynchronous operation results # Async, when Func is registered to a process, the program continues to execute down, returning an object that has a GET method to fetch the value # Obj.get () will block, Know that the corresponding Func execution is complete to get the result. Requires close and join to maintain code synchronization for multiple processes and main processes # asynchronous processing of tasks, all processes in the process pool are daemons # There are callback functions callback# 4. P.close (): Closes the process pool and prevents further action. If all operations persist, they will wanc# 5 before the worker process terminates. P.join (): Waits for all worker processes to exit. This method can only be called after close () or Teminate () # 1. The return value of Method Apply_async () and Map_async () is an instance of Asyncresul, obj # 2.obj . Get (): Returns the result and, if necessary, waits for the result to arrive. Timeout is optional. If #在指定时间内还没有到达, an exception is thrown. If an exception is thrown in a remote operation, it will be thrown again when this method is called # 3.obj.ready (): If the call is complete, return True # 4.obj.successful (): If the call is complete and no exception is thrown, it will be referenced again when this method is called # 5.obj.wait ([timeout]): Wait for the result to become available # 6.obj.terminate (): Immediately terminates the There are worker processes that do not perform any cleanup or end of any pending work. If P is garbage collected #将自动调用此函数 # P.MAP process pool and process efficiency test # from multiprocessing import pool,process# import os,t ime## def func (num): # num + = # print (num) # # if __name__ = = ' __main__ ': # p = Pool (5) # Number of processes in open process Pool 5 # s Tart = Time.time () # Task start in Process Pool # P.map (func,[i For I in range (100)]) # The Map Method # P.close () # In the process pool is not allowed to add a task to the process Pool # p.join () # Wait for all the processes in the process pool to finish All Tasks # pri NT ("Efficiency of the task in the process pool:", Time.time ()-start) # # start = Time.time () # Normal process start Timer # p_l = []# for i in range (100): # P1 = Process (target=func,args= (i,)) # P1.start () # P_l.append (p1) # [P1.join () for P1 in p_l]# Prin T ("Multi-Process task efficiency:", Time.time ()-start) # #p. Apply () refers to the process in the process pool to help you do the task # #p. Apply_async () is to let the process in the process pool, asynchronous to help you do the task # The result is a process pool task 0.13 multi-process task 5.1# Process pool Synchronous Call # from multiprocessing import pool,process# import time## def func (num): # num + = # Return num# def func1 (NUM1): # NUM1 = # return num1## if __name__ = = ' __main__ ': # p = Pool (5) # start = Ti Me.time () # for I in range: # Ret1 = p.apply (func1,args= (i,)) # ret = p.apply (func,args= (i,)) # Simultaneous processing of 100 tasks, synchronous means no matter how many tasks are one into # print (RET,RET1) # # Print (Time.time ()-start) # Process pool synchronous call two # from multiprocessing I Mport pool# Import Time,os## def work (n): # receive parameter # Print for I ('%s run '%os.getpid ()) # Get pid# time.sleep (3) # return n**2## if __name_ _ = = ' __main__ ': # p = Pool (5) # Process pool open 5 Processes # ret_l = []# for i in range: # # res = p.apply (work,args= (I,)) # Synchronous call until this task is completed get res, Task # Print (res) # Ret_l.append (res) # #work执行的过程中可能有阻塞也有可能没有阻塞 # #但不管该 If a task is blocked, the synchronous call waits in place # print (ret_l) # # # Process Pool Asynchronous Processing Task # from multiprocessing import pool# import time# def func (num): # nu M + = # return num## if __name__ = = ' __main__ ': # p = Pool (5) # start = Time.time () # LST = []# for i In Range: # # ret = P.apply_async (func,args= (i)) # Asynchronous processing of 100 tasks, asynchronous values, five processes in the process pool, five tasks can be processed # #print (ret) # Returns the process pool object address via Get function-specific value # print (Ret.get ()) # lst.append (ret) # p.close () #关闭 # p.join () # #print (LST) # for i in lst:# New_ret = I.get () # Get results by Get # #print (new_ret) # Print (Time.time ()-start) # # Process pool Asynchronous Processing Task two #From multiprocessing import pool# import time,os## def work (n): # print ('%s run '%os.getpid ()) # return n**2## if __ name__ = = ' __main__ ': # p = Pool (5) # Process pool open five processes # ret_lst = []# for i in range: # ret = P.apply_asyn C (work,args= (i)) # runs asynchronously, depending on the number of processes in the process pool, each # #最多执行5个子进程在异步执行 # #返回结果加入列表, the process is returned, and then a new task # #注意: Five processes in the process pool do not will open at the same time or end at the same time # #而是执行完一个就释放一个进程, this process to receive a new task # ret_lst.append (ret) # #异步apply_async用法: If you use the asynchronous submit task, the master needs to use the join , wait for the # #任务都处理完 in the process pool, and then you can collect the results with get # #否者, the main process ends, the process pool may not have time to execute, and then end with the main process # p.close () # P.join () # for I in ret_l st:# New_ret = I.get () # Gets the result # #使用get来获取apply_async的结果, if it is apply, there is no Get method # #因为apply是同步执行, get the result immediately, also root This does not require get# print (new_ret) # Process pool synchronous execution and asynchronous execution efficiency comparison # from multiprocessing import pool# import time## def func (num): # nu M + = # return num### if __name__ = = ' __main__ ': # p = Pool (5) # start = Time.time () # for I in range (10000): # ret = p.apply (Func,args= (i,)) # Process pool Synchronous Execution # print (' Process pool ' YoY execution time: ', Time.time ()-start) # # start = Time.time () # for I in range (100 XX): # Ret1 = P.apply_async (func,args= (i)) # Process pool Asynchronous Execution # p.close () # p.join () # print ("Process pool asynchronous Execution time:", time.t IME ()-start) # Result: The process pool performs the same amount of tasks, asynchronously executes the Block # callback function #需要回调函数的场景: Once any of the tasks in the process pool have been processed, inform the process #我好了饿 immediately, and you can process my results. The main process falls off a function to handle the As a result, the function that functions #我们可以把耗时间 (blocked) is put into the process pool and then specifies the callback function (the main process complex execution) #这样主进程在执行回调数时就省去I/O process, directly to the result of the task #如果在主进程中等待进程池中所有任务都执行完毕后, Once the result is processed uniformly, no callback function is required
Python Process Pool