Day-4 Python multi-process programming knowledge points Summary

1, Python Multi-process introduction

Due to the limitations of Python design (I'm talking about the CPython we used). You can use up to 1 CPU cores. Python provides a very useful multi-process package multiprocessing, he provides a set of interfaces similar to multithreading, with start, run, and so on, we just need to define a function, Python will do everything else for us. With this package, you can easily convert from single-process to concurrent execution.

2. Precautions

A) on UNIX platforms, when a process is terminated, the process needs to be called by its parent process wait, or the process becomes a zombie process (Zombie). Therefore, it is necessary to call the join () method on each Process object (which is actually equivalent to wait). For multithreading, this is not necessary because there is only one process.

b) multiprocessing provides an IPC (such as pipe and queue) that is not available in the threading package, and is more efficient. pipe and queue should be given priority, avoiding the use of synchronization methods such as Lock/event/semaphore/condition (because they do not occupy the resources of the user process).

c) Multiple processes should avoid sharing resources. In multi-threading, we can share resources more easily, such as using global variables or passing parameters. In a multi-process scenario, the above method is not appropriate because each process has its own independent memory space. At this point we can share the resources by sharing the memory and the manager's method . However, this improves the complexity of the program and reduces the efficiency of the program because of the need for synchronization.

3. Common interface

Event (): The events for the process are used for the main thread to control the execution of other processes, and the events provide three methods wait, clear, set

Queue (): Queues for processes, providing get and put methods

Process (): Create a new process

Lock (): Process lock

Semaphore: A process synchronization mechanism with counting, when the release is called, increments the calculation when acquire, reduces the count when the count is 0 o'clock, automatically blocks, waits for release to be called

Pipe (): Create a process bidirectional pipeline

Manager (): A more advanced multi-process communication method that supports any data structure supported by Python, without limiting whether multiple processes originate from the same parent process

Lock (): Process lock

Pool (): You can provide a specified number of processes for the user to invoke, and when a new request is submitted to the pool, a new process is created to execute the request if it is not full, but if the number of processes in the pool has reached the specified maximum, the request waits until the process ends in the pool. To create a new process to it.

Condition (): Condition is called a conditional variable and provides the wait and notify methods in addition to the acquire and release methods that are similar to lock.

　　

4. code example

" "learn multi-threaded communication: Queue,pipe,manage,event" "ImportMultiprocessingImport Time#loop fetching data from the process queuedefProceedataget (q,p,parent_event,child_event,manage_d,manage_l): forIinchRange (10):        Print(Q.get ()) forIinchRange (20,30): P.send (i) Parent_event.set () child_event.wait () forIinchRange (20,30):        Print("B Process:"+Str (P.RECV ())) manage_d["1"] = 1Manage_l.append ("2")#looping from the process queue to write datadefProceedataput (q,p,parent_event,child_event,manage_d,manage_l): forIinchRange (10): Q.put (i) forIinchRange (40,50): P.send (i) Child_event.set () parent_event.wait () forIinchRange (20,30):        Print("A Process:"+Str (P.RECV ())) manage_d["2"] = 2Manage_l.append ("2")if __name__=="__main__": Start_time=time.time ()#Define a process queueQ =multiprocessing. Queue ()#Define a process bidirectional pipelineParent_conn,child_conn =multiprocessing. Pipe ()#define two process eventsParent_event =multiprocessing. Event () child_event=multiprocessing.    Event () multiprocessing. #Define a single Manager objectManage =multiprocessing. Manager () manage_d=manage.dict () manage_l=manage.list ()#define two processesL =[] P1= multiprocessing. Process (target = proceedataget,args=(q,child_conn,parent_event,child_event,manage_d,manage_l)) P1.start () L.append (p1) P2= multiprocessing. Process (target = proceedataput,args=(q,parent_conn,parent_event,child_event,manage_d,manage_l)) P2.start () l.append (p2)#wait for the process to finish executing     forP_listinchl:p_list.join () end_time=time.time ()Print(manage_d)Print(manage_l)Print("mutiple proccess Cost:%d"% (End_time-start_time))

View Code

Reference Link:https://docs.python.org/2/library/multiprocessing.html

Day-4 Python multi-process programming knowledge points Summary