13. Multi-process multiprocessing, process pool

Content Related:

Multiprocessing:

• Creation and operation of processes
• Process common related functions

Process Pool:

• Why do you have a process pool
• Process pool creation and operation: serial, parallel
• callback function

Multi-process multiprocessing:

Multiple processes in Python require the use of the multiprocessing module

• Creation and operation of multiple processes:

1. Process creation: Process object =multiprocessing. The Process (target= function name, args= (parameter,)) "complements that because args is a tuple, a single parameter is added", ""

2. Process run: Process object. Start ()

The join of the process is the same as the join of the thread, meaning " block the current process until the process that calls the Join method finishes executing, and then proceeds to the current process"

Note: This must be used in code in Windows, which is not required in Linux

ImportMultiprocessing,time,osdefThread_run ():Print(Threading.current_thread ())defRun (name): Time.sleep (1)    Print("Hello", Name,"Run in", Os.getpid (),"Ppid:", Os.getppid ())if __name__=='__main__':#must be addedobj=[]     forIinchRange (10): P=multiprocessing. Process (target=run,args= ('Bob',)) Obj.append (P) p.start () start_time=time.time () forIinchObj:i.join ()Print("Run in main")    Print("spend time:", Time.time ()-start_time)

• Same as Multithreading: You can also create processes by inheriting the multiprocessing process

The class that inherits multiprocessing's process class has two main things to do:

1. If you initialize your own variables, first call the parent class's __init__ () "If not called, you have to fill in the relevant parameters, trouble!" "Then do your own initialization, if you do not need to initialize your own variables, then you do not need to rewrite __init__, directly using the parent class __init__ can" inherit "

2. Overriding the Run function

Import Multiprocessing class myprocess (multiprocessing. Process):    def  Run (self):        print("run in myprocess  ")if__name__= ="__main__":    p =myprocess ()    P.start ()    p.join ()

Process common related functions:

• Os.getpid (): Gets the current process number.
• Os.getppid (): Gets the parent process number of the current process.
• The Process object. Is_alive (): Determine if the process is alive

• The Process object. Terminate (): End Process "Not recommended method, reality less"

Process Pool:

• Why process pooling is required
• If you want to start a large number of child processes, you can create the child processes in bulk using the process pool, and the process pool can limit the number of processes that are running "too many people want to swim, and the pool capacity determines the number of people swimming.
• The pool class can provide a specified number of processes for the user to call, and when a new request is submitted to the pools, a new process will be created to execute the request if it is not full. If the process pool is full, the request tells you to wait until the process ends in the pool before a new process is created to execute the request

• Process pool creation and use:
• Use process pool to import: From multiprocessing Import pool
• Create process Pool: Process Pool object =pool (capacity)
• To add a process to a process pool:
• Serial: Process Pool object. Apply (func= function name, args= (parameter,))

 fromMultiprocessingImportPoolImportTime,osdeffunc1 (i): Time.sleep (1)    Print("run in Process:", Os.getpid ())if __name__=="__main__": Pool=pool (5) Start_time=time.time () forIinchRange (10): Pool.apply (func=func1,args= (i,))#Serial, here is the plus one run out plus onePool.close ()#Close before waitingPool.join ()Print("main run Done,spend_time:", Time.time ()-start_time)

• Parallel: Process Pool object. Apply_async (func= function name, args= (parameter,), callback= callback function)

 fromMultiprocessingImportPoolImportTime,osdeffunc1 (i): Time.sleep (1)    Print("run in Process:", Os.getpid ())if __name__=="__main__": Pool=pool (5) Start_time=time.time () forIinchRange (10): Pool.apply_async (func=func1,args= (i,))#Parallelpool.close ()#Close before waitingPool.join ()Print("main run Done,spend_time:", Time.time ()-start_time)#2.6, proving to be parallel

• Use of callback functions: In parallel, the callback= callback function is supported, and the callback function is called when a process finishes executing, and the parameter is the return value in Func

 fromMultiprocessingImportPoolImportTime,osdeffunc1 (i): Time.sleep (1)    Print("run in Process:", Os.getpid ())return "filename"defLog (ARG):##参数为进程创建中func的函数的返回值    Print("Log Done:", Arg)if __name__=="__main__": Pool=pool (5) Start_time=time.time () forIinchRange (10): Pool.apply_async (func=func1,args= (i,), Callback=log,)#The log parameter is the return value of Func1pool.close ()#Close before waitingPool.join ()Print("main run Done,spend_time:", Time.time ()-start_time)

• Note: Pool calling a method on an object join() waits for all child processes to complete before the call must be called before the call join() close() close() can continue to add new Process . "Meaning is like the swimming pool only sell 1 hours of tickets, agreed to close 5 o'clock, then 4 points after more can not sell tickets, has been waiting for the pool inside the people out and then closed, the process pool close is a closing meaning, not the end of the meaning, it just closed the door, and the process can also run" " The join of the process pool is the execution of all the processes in the pond, and if it comes back in the process, it will be endless, so you need to close the entry and wait for the process to end.
• Small test:

13. Multi-process multiprocessing, process pool