Multiprocessing process-based "line program" interface python

multiprocessing is applied in the python2.6+ version.

Multi meaning is multiple, processing means process also is to realize multi-process. You can implement multithreading with multiple CPUs that call your computer.

Multiprocessing is a package that supports the use of API generation processes like the threading module. The multiprocessing package provides both local and remote concurrency, effectively avoiding global interpreter locks by using child processes rather than threads. Therefore, the multiprocessing module allows programmers to make full use of a given machine's multiple processors. It can be allowed on both UNIX and Windows.

The following mainly describes the process classes :

A thread is produced by creating a process object and then invoking the start () method. The process inherits the Threading.thread API. Now give an example of multiprocess:
To create a process by using process:

fromimport Processdef f(name):    print‘hello‘, nameif‘__main__‘:    p = Process(target=f, args=(‘bob‘#创建一个线程    #调用start方法 启动    p.join()  

Here is an example of a live thread ID:

 fromMultiprocessingImportProcessImportOs def info(title):    PrintTitlePrint ' module name: ', __name__ifHasattr (OS,' Getppid '):# can only be used in Unix        Print ' parent process: ', Os.getppid ()Print ' process ID: ', Os.getpid () def F(name):Info' function f ')Print ' Hello ', nameif__name__ = =' __main__ ': info (' main Line ') p = Process (Target=f, args= (' Bob ',)) P.start () P.join ()

The code uses the OS's getpid () to get the ID of the parent process using GetIP () to get the ID of the current process, and also to use Getppid to get the parent process ID of the parent process.

IPC (inter-process communication) inter-process communication

The multiprocessing package has Pipe and Queue classes to support these two IPC mechanisms, respectively. Pipe and queue can be used to transfer common objects.
The Pipe can be unidirectional (Half-duplex) or bidirectional (duplex). We passed the mutiprocessing. Pipe (Duplex=false) creates a one-way pipe ( default is bidirectional ). A process enters an object from the pipe end and is then received by the process on the other end of the pipe, and the one-way pipeline allows only the process input at the end of the pipe, while the bidirectional pipe allows input from both ends.
Example: (One-way)

 fromMultiprocessingImportProcess, Pipe def F(conn): #conn为父线程传递过来的pipe对象Conn.send ([ the,None,' Hello '])#在pipe对象的一端发送数据Conn.close ()#关闭if__name__ = =' __main__ ': parent_conn, Child_conn = Pipe ()#定义两个Pipe对象来创建p = Process (Target=f, args= (Child_conn,))#创建一个进程 passed parameters are pipe objectsP.start ()PrintPARENT_CONN.RECV ()# prints "[All, None, ' hello ']" #在父进程中使用另一个pipe对象的recv () method accepts dataP.join ()

Each connection object has a send and recv method, and it is important to note that if two processes or threads simultaneously read or write to the terminal of the pipe object, the terminal may be caused. There is no risk if you use different terminals of the pipe at the same time.

The following example: (bidirectional)

ImportMultiprocessing asMul def proc1(pipe):Pipe.send (' Hello ') Print (' Proc1 rec: ', Pipe.recv ()) def proc2(pipe):Print' proc2 rec: ', Pipe.recv ()) Pipe.send (' Hello, too ')if__name__ = =' __main__ ': Mul.freeze_support () pipe1,pipe2= mul. Pipe ()#创建两个pipe对象    # Pass An end of the pipe to process 1P1 = Mul. Process (Target=proc1, args= (Pipe1,))#创建p1线程    # Pass The other end of the pipe to process 2P2 = mul. Process (TARGET=PROC2, args= (Pipe2,))#创建p2线程P1.start ()#调用线程1P2.start ()#调用线程2P1.join ()#这里等待线程1执行完成P2.join ()#等待线程2执行完成# Output: (' proc2 rec: ',' Hello ')(' Proc1 rec: ',' Hello, too ')

This example enables communication between two threads, enabling two-way communication. Thread 1:proc1 sends a "hello" string in thread proc2, the "Hello" string that is sent in the receive thread 1proc1 and outputs, and then PROC2 sends a "Hello, too" string, thread 1 Proc1 receives the string sent by PROC2 and outputs.

Another way to implement IPC is described below:
Queue:
A queue is similar to a pipe, and is a first-in-one-out structure. But the queue allows multiple processes to be put in, and multiple processes are fetching objects from the queue. The queue is created using Mutiprocessing.queue (MAXSIZE), which maxsize represents the maximum number of objects that can be held in a queue.
The following program shows the use of the queue:

fromimport Process, Queuedef f(q):    q.put([42None‘hello‘])  #调用主函数中p进程传递过来的进程参数 put函数为向队列中添加一条数据。if‘__main__‘:    #创建一个Queue对象    #创建一个进程    p.start()    print q.get()    # prints "[42, None, ‘hello‘]"    p.join()

Above is a simple application of the queue, using the queuing Q object to call the Get function to get the data that was first entered in the queue.
To give a more complicated example:

Import Multiprocessingimport Time# functions to enter data into a queuedef inputq (queue): info = str (os.getpid ()) +' (Put): '+ STR ( Time. Time()) queue.put(info)# functions to output data to a queuedef outputq (queue,lock): info = queue.Get() Lock.acquire () print (str (os.getpid ()) +' (GET): '+ info) lock.release ()# Mainif__name__ = =' __main__ ': Multiprocessing.freeze_support () record1 = []# Store Input ProcessesRecord2 = []# Store Output ProcessesLock = multiprocessing. Lock ()#使用锁方法输出错乱Queue = multiprocessing. Queue (3)# Input Process     forIinchRangeTen):Process= multiprocessing. Process (target=inputq,args= (queue,))Process. Start () Record1.append (Process)# Output Process     forIinchRangeTen):Process= multiprocessing. Process (target=outputq,args= (Queue,lock))Process. Start () Record2.append (Process) forPinchRecord1:p.join () queue.Close()# Close the queue if no data enters the queue     forPinchRecord2:p.join ()#输出结果为:15212(Get):7180(put):1440649672.228820(Get):7020(put):1440649672.856164(Get):7872(put):1440649673.3910904(Get):9436(put):1440649673.685016(Get):16840(put):1440649674.0712004(Get):12848(put):1440649674.3814456(Get):11608(put):1440649674.662468(Get):16884(put):1440649674.856464(Get):8944(put):1440649674.897576(Get):15476(put):1440649674.92

In this example, 20 processes are created, 10 are used to write to the queue data, and 10 are used to output the data in the queue.

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Multiprocessing process-based "line program" interface python