Python process programming

multi-Process multiprocess module

multiprocessing  is A package, supports spawning processes using an API similar to The threading  module. The multiprocessing   Package offers both local and remote concurrency, effectively side-stepping The global interpreter Lock  by using subprocesses instead of THR Eads. Due to this, The  multiprocessing  module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.

The syntactic structure of this module is basically similar to the threading module.

Find a multi-process blog connection: https://www.cnblogs.com/Mr-Murray/p/9026955.html

 fromMultiprocessingImportProcessImport TimeImportOSdefinfo ():Print("\033[35;1mthe Time is%s\033[0m"%time.ctime ())Print("The parent is%s; the child process is%s"%(Os.getppid (), Os.getpid ()))if __name__=="__main__":     forIinchRange (10): P= Process (target=info) P.start () P.join ()

There is no difference between this code and the multithreading previously created with the threading module, but if executed on a Windows system, you must add if __name__ = = "__main__": statement for the following reasons:

In the Windows operating system because there is no fork (the mechanism of creating a process in the Linux operating system), when the child process is created, it will automatically import the file to launch it,
In the import, the entire file was executed. Therefore, if the process () is written directly in the file, it will be infinitely recursive to create the child process error. So you have to create the child
The part of the process uses if __name__ = = ' __main__ ' to judge the protection, when the import  , it will not run recursively.

To generate a process in an instantiated manner:

ImportOS, time fromMultiprocessingImportProcessclassmyprocess (Process):def __init__(self): Super (myprocess, self).__init__()    defRun (self):Print("\033[35;1mthe Time is%s\033[0m"%time.ctime ())Print("The parent is%s; the child process is%s"%(Os.getppid (), Os.getpid ()))if __name__=="__main__": P=myprocess () P.start ()
#在调用p. Start, the Run method in the class is automatically executed, and the Run method is essential. The same as the Run method in the thread in the threading class. 

The daemon in the process is the same as the daemon thread in the threading module, and the child process ends when the main process ends.

ImportOS, time fromMultiprocessingImportProcessdeffun1 ():Print("Starting%s". Center (50,"-") %time.ctime ()) Time.sleep (3)    Print("Stopping%s". Center (50,"-") %time.ctime ())if __name__=="__main__": P= Process (target=fun1) P.daemon=True P.start () time.sleep (1)--------------------------------------------------------------------------The child process is to perform 3s, but only 1s is executed in the main process, after the daemon is set, the main process ends, regardless of the state of the child process, the child process exits

Use multi-process to realize simple socket concurrent connection.

Server-side code:Importsocket, time fromMultiprocessingImportProcessdefhandle (CONN,ADDR):Print("The connection is from %s at%s Port"%addr) conn.send (b"Hello World") Data= CONN.RECV (1024) conn.send (data)if __name__=="__main__": HOST="localhost"PORT= 51423ADDR=(HOST, PORT) s=Socket.socket (socket.af_inet, socket. SOCK_STREAM) S.bind (ADDR) S.listen (5)     whileTrue:conn, addr=s.accept () p= Process (Target=handle, args=(CONN,ADDR)) # The data between processes is not shared, so you need to pass the coon as a parameter P.start () S.close () client-side code:ImportSockethost="localhost"PORT= 51423ADDR=(HOST, PORT) s=Socket.socket (socket.af_inet, socket. SOCK_STREAM) S.connect (ADDR) data= S.RECV (1024)Print(Data.decode ()) whileTrue:info= Input (">>>:") S.send (Info.encode ()) Data= S.RECV (1024)    Print(Data.decode ())

Attention:

Data between processes cannot be accessed from each other, so Conn must be passed as a parameter.

What should I do with communication between processes?

queues and pipelines in a process:

 fromMultiprocessingImportProcess, Queue fromTimeImportSleepImportRandom#use the process to write a build consumer model,#The generator inserts a random number into the queue, and the consumer takes out a value in the queuedefProductor (q): whileTrue:sleep (0.3)        Print("store an num") Q.put (Random.randint (1,100))defConsumer (q): whileTrue:sleep (0.2)        Print("Getting an num") Q.get ()if __name__=="__main__": Q=Queue () Proc1= Process (Target=productor, args=(q,)) Proc2= Process (Target=consumer, args=(q,)) Proc1.start () Proc2.start ( )

This is just a simple model to illustrate the queue's communication between processes.

The queue usage here is basically the same as the queue usage in threads, just one for process communication and one for thread communication.

Pipeline

Simple and practical pipeline:

 from  multiprocessing import   Process, Pipe  def   f (conn): Conn.send (  " hello World  Span style= "COLOR: #800000" > " ) conn.close ()  if  __name__  = =  "  __main__   " : Parent_conn, Child_conn  = Pipe () p  = Process (target=f, Args= (child_ conn,)) P.start ()  print   (parent_conn.recv ()) P.join ()  

Manager

Manager () returns the manager object that controls a server process that contains Python objects that can be accessed by other processes through proxies. So as to achieve inter-process data communication and security.

A Manager object returned by Manager() controls a server process which holds Python objects and allows other processes to Mani Pulate them using proxies.

A Manager returned byManager()Would support typeslist,dict,Namespace,Lock,RLock,Semaphore,BoundedSemaphore,Condition,Event,Barrier,Queue,ValueandArray.

A simple and practical example:

 fromMultiprocessingImportProcess, ManagerImportRandomdeff (list): List.append (random.randint (0,100))    Print(list)if __name__=="__main__": P_list=[] with Manager () as Manager:l=manager.list () forIinchRange (10): P= Process (target=f,args=(L,)) P.start () P_list.append (p) forResinchP_list:res.join ()

Each process adds a data to the list and executes the following results:

[95]
[95, 25]
[95, 25, 31]
[95, 25, 31, 70]
[95, 25, 31, 70, 9]
[95, 25, 31, 70, 9, 17]
[95, 25, 31, 70, 9, 17, 96]
[95, 25, 31, 70, 9, 17, 96, 71]
[95, 25, 31, 70, 9, 17, 96, 71, 96]
[95, 25, 31, 70, 9, 17, 96, 71, 96, 3]

Process Pool:

A process sequence is maintained internally by the process pool, and when used, a process is fetched in the process pool, and the program waits until a process is available in the process pool sequence if there are no incoming processes available for use.

 fromMultiprocessingImportProcess, PoolImport TimedefFoo (i): Time.sleep (2)    returni + 100defBar (ARG):Print('-->exec Done:', Arg)if __name__=="__main__": Pool= Pool (5)     forIinchRange (10): Pool.apply_async (func=foo, args= (i,), callback=Bar)#pool.apply (Func=foo, args= (i,))    Print('End') Pool.close () Pool.join ()

Note: The callback callback function is used when calling Apply_async

Process pool detailed, see a blog: https://www.cnblogs.com/qiangyuge/p/7455814.html

Python process programming