Process pools and threads are initially

1. Process Pool

2. Threads
# theory
# Threading.thread to create threads

1. Process Pool

Why do you have a process pool? The concept of a process pool. During a program's actual process, there are thousands of tasks that need to be performed during busy hours, and there may be only sporadic tasks at leisure. So when thousands of tasks need to be executed, do we need to create thousands of processes? First, the creation process consumes time, and the destruction process also consumes time. Second, even if thousands of processes are turned on, the operating system does not allow them to execute simultaneously, which can affect the efficiency of the program. Therefore, we cannot open or end the process indefinitely according to the task. So what are we going to do about it? Here, to introduce the concept of a process pool, define a pool, put a fixed number of processes inside, there is a need to come, take a pool of processes to handle the task, wait until the process is finished, processes are not closed, but the process is put back into the process pool to continue to wait for the task. If there are many tasks that need to be performed, the number of processes in the pool is not enough, and the task waits for the previous process to complete before returning to the idle process to continue execution. That is, the number of processes in the pool is fixed, and at most a fixed number of processes are running at the same time. This will not increase the operating system scheduling difficulty, but also save the opening and closing process time, but also to a certain extent, to achieve concurrency effect.

Parameter introduction

Pool ([numprocess  [, initializer [, Initargs]]): Create a process pool

1 Numprocess: The number of processes to be created and, if omitted, the value of Cpu_count () will default to 2 initializer: The callable object to execute when each worker process starts, default to None3 Initargs: is the parameter group to pass to the initializer

Main methods

P.apply (func [, args [, Kwargs]): Executes func (*args,**kwargs) in a pool worker process and returns the result. It should be emphasized that this operation does not execute the Func function in all pool worker processes. If you want to execute the Func function concurrently with different parameters, you must call the P.apply () function from a different thread or use P.apply_async () '  P.apply_async (func [, args [, Kwargs]): Executes func (*args,**kwargs) in a pool worker process and returns the result. The result of this method is an instance of the AsyncResult class, and callback is a callable object that receives input parameters. When the result of Func becomes available, the understanding is passed to callback. Callback does not prohibit any blocking operations, otherwise it will receive results from other asynchronous operations. "'     p.close (): Close the process pool to prevent further action. If all operations persist, they will complete p.jion () before the worker process terminates  : waits for all worker processes to exit. This method can only be called after close () or teminate ()

Other methods (extended understanding)

The return value of Method Apply_async () and Map_async () is an instance of Asyncresul obj. The instance has the following method Obj.get (): Returns the result and waits for the result to arrive if necessary. Timeout is optional. If it has not arrived within the specified time, a one will be raised. If an exception is thrown in a remote operation, it is raised again when this method is called. Obj.ready (): Returns True if the call is complete obj.successful (): Returns True if the call finishes without throwing an exception, or if this method is called before the result is ready, throws an exception obj.wait ([timeout]): Waits for the result to become available. Obj.terminate (): Immediately terminates all worker processes without performing any cleanup or end of any pending work. If P is garbage collected, this function is called automatically

If you use the P.apply () method, this becomes synchronous.

Examples are as follows:

 import   OS  import   time  from  multiprocessing import   pool,process  def   Fun (): 
    print  (666 0.5)  if  __name__  = =  " __main__   '  

Only use P.apply_async () to have an asynchronous effect:

ImportOSImport Time fromMultiprocessingImportpool,processdefFun ():Print(666) Time.sleep (0.5)if __name__=='__main__': P= Pool (4)     forIinchRange (20): P.apply_async (Fun) p.close ()#shutting down the process pool is not working. Instead, the process pool is closed so that the task cannot continue to commitP.join ()#wait for the task that is submitted in this pool to finish executing. Indicates that the code in all child processes has finished executing the main process

If the function has a return value, then how to get it in the process pool, refer to the following:

ImportOSImport Time fromMultiprocessingImportpool,processdefFun (i):Print(666) Time.sleep (0.5)    returnIif __name__=='__main__': P= Pool (4) P_lst= []     forIinchRange (20): Ret= P.apply_async (fun,args=(i,)) P_lst.append (ret) p.close () P.join () [Print(I.get ()) forIinchP_lst]#asynchronous Commit, get the return value, once a task has finished executing it can get a result (order is in the order of submitting the Task)#Async-Apply_async#1. If the commit task is asynchronous, then the process pool and the main process are asynchronous after the task is committed ,    #The main process does not automatically wait for tasks in the process pool to complete#2. If you need to wait for the main process, P.join    #but the behavior of join is dependent on close#3. If the function has a return value    #The return value can also be obtained by Ret.get ().    #But if one side commits to get the return value, the program becomes synchronized.    #so to keep the asynchronous effect, it should be said that the return object is saved in the list, all the tasks submitted after the completion of the results    #this way you can also remove the join to complete the blocking of the main process the task in the pool is completed

# asynchronous Apply_async
# 1. If the commit task is asynchronous, then the process pool and the main process are also asynchronous after the task is committed.
#主进程不会自动等待进程池中的任务执行完毕
# 2. If you need to wait for the main process, p.join
# But the behavior of join is dependent on close
# 3. If the function has a return value
# You can also get the return value by Ret.get ()
# But if one side commits to get the return value, the program becomes synchronized.
# So to keep the asynchronous effect, it should be said that the return object is saved in the list, all the tasks submitted after the completion of the results
# This way you can also remove a join to complete the blocking of the main process the task in the pool is completed

Callback functions in the process pool

scenario where a callback function is required: once any of the tasks in the process pool have been processed, inform the main process immediately: I'm all right, you can handle my results. The main process calls a function to process the result, which is the callback function we can put the time-consuming (blocking) task into the process pool, and then specify the callback function (the main process is responsible for execution), so that the main process in the execution of the callback function of the process of eliminating the I/O, directly get the results of the task.

# callback Function _ executes in the main process # Specify the callback parameter when initiating a task # after each process finishes executing the Apply_async task, the return value is passed directly as a parameter to the callback function, executing the code in the callback function

Example code:

ImportOs,time fromMultiprocessingImportPooldefFun (i):Print("PID process is%s,num is%s"%(Os.getpid (), i))returnIdefCall_back (i):Print("This is callback", i) time.sleep (0.5)if __name__=='__main__': P= Pool (4)     forIinchRange (10): P.apply_async (func=fun,args= (i,), callback=call_back) P.close () P.join ()

Process pools and threads are initially