36 Multi-process

To get the Python program to implement multi-process (multiprocessing), we first understand the operating system knowledge.

The Unix/linux operating system provides a fork() system call, which is very special. A normal function call, called once, is returned once, but fork() called once, and returned two times, because the operating system automatically replicates the current process (called the parent process) with a copy (called a child process), and then returns within the parent and child processes, respectively.

The child process returns forever 0 , and the parent process returns the ID of the child process. The reason for this is that a parent process can fork out many child processes, so the parent process has to note the ID of each child process, and the child process only needs to invoke getppid() the ID of the parent process.

Python's os modules encapsulate common system calls, including fork the ability to easily create child processes in a Python program:

 import osprint (  " process (%s) Start ...   "% Os.getpid ()) # only works on Unix /linux /mac:pid  = Os.fork ()  if  pid = = 0  : Print (  '  

The results of the operation are as follows:

Process (876) Start ... I (876) just created a child process (877). I am Child process (877is876.

forkThe above code cannot be run on Windows because Windows is not called. Because the Mac system is based on the BSD (Unix) kernel, so, running under the Mac is no problem, we recommend that you learn python! with Mac

With a fork call, a process can replicate a child process to handle a new task when it receives a new task, and the common Apache server is the parent process listening on the port, and whenever there is a new HTTP request, fork out the child process to process the new HTTP request.

Multiprocessing

If you are going to write a multi-process service program, Unix/linux is undoubtedly the right choice. Because Windows didn't fork call, wouldn't it be possible to write multi-process programs in Python on Windows?

Because Python is cross-platform, nature should also provide a cross-platform, multi-process support. multiprocessingmodules are multi-process modules with cross-platform versions.

multiprocessingThe module provides a Process class to represent a process object, and the following example demonstrates starting a child process and waiting for it to end:

 frommultiprocessing Import Processimport os# child process to execute the Code def RUN_PROC (name): Print ('Run Child process%s (%s) ...'%(name, Os.getpid ()))if__name__=='__main__': Print ('Parent process%s.'%os.getpid ()) P= Process (Target=run_proc, args= ('Test',)) Print ('Child process would start.') P.start () P.join () print ('Child process end.')

The results of the implementation are as follows:

928 . Process would start. Run Child process Test (929) ... Process end.

When you create a child process, you only need to pass in a parameter that executes functions and functions, create an Process instance, and start it with a start() method, so that the process is fork() simpler to create.

join()Method can wait for the child process to finish before continuing to run down, typically for inter-process synchronization.

Pool

If you want to start a large number of child processes, you can create the child processes in batches using the process pool:

 frommultiprocessing Import Poolimport OS, time, Randomdef long_time_task (name): Print ('Run Task%s (%s) ...'%(name, Os.getpid ())) Start=time.time () time.sleep (Random.random ()*3) End=time.time () print ('Task%s runs%0.2f seconds.'% (name, (End-start )))if__name__=='__main__': Print ('Parent process%s.'%os.getpid ()) P= Pool (4)     forIinchRange5): P.apply_async (long_time_task, args=(i,)) print ('waiting-subprocesses done ...') P.close () P.join () print ('All subprocesses is done .')

The results of the implementation are as follows:

Parent process669. Waiting forAll subprocesses-Done ... Run Task0(671)... Run Task1(672)... Run Task2(673)... Run Task3(674)... Task2Runs0.14seconds. Run Task4(673)... Task1Runs0.27seconds. Task3Runs0.86seconds. Task0Runs1.41seconds. Task4Runs1.91seconds. All subprocesses is done.

Code interpretation:

Invoking a method on an Pool object join() waits for all child processes to complete before the call must be called before the call join() close() close() can continue to add new Process .

Note that the result of the output, task 0 ,, 1 2 3 is executed immediately, and the task 4 waits for a previous task to complete before it executes, because Pool the default size on my computer is 4, so Execute up to 4 processes at a time. This is a Pool deliberately designed limitation, not the operating system's limit. If you change to:

p = Pool (5)

You can run 5 processes at a time.

Because Pool the default size is the number of cores of the CPU, if you unfortunately have a 8-core CPU, you have to submit at least 9 sub-processes to see the wait effect above.

Child process

Many times, a child process is not itself, but an external process. After we have created the child process, we also need to control the input and output of the child process.

subprocessThe module allows us to start a subprocess very conveniently and then control its input and output.

The following example shows how to run a command in Python code nslookup www.python.org , which works just like the command line:

Import subprocessprint ('$ nslookup www.python.org'= Subprocess.call ([ ' nslookup ' ' www.python.org ' ]) print ('Exit code:', R)

Operation Result:

$ nslookup www.python.orgServer:         192.168. 19.4 Address:     192.168. 19.4#Non-authoritative answer:www.python.org    =  Python.map.fastly.net.Name:    199.27. 79.223  0

If the child process also needs input, it can be communicate() entered by method:

Import Subprocessprint ('$ nslookup') P= subprocess. Popen (['nslookup'], stdin=subprocess. PIPE, Stdout=subprocess. PIPE, stderr=subprocess. PIPE) output, err= P.communicate (b'Set q=mx\npython.org\nexit\n') Print (Output.decode ('Utf-8')) Print ('Exit Code:', P.returncode)

The above code is equivalent to executing the command at the command line and nslookup then manually entering:

set q=mxpython.orgexit

The results of the operation are as follows:

$ nslookupserver:192.168.19.4Address:192.168.19.4# -Non-authoritative answer:python.org mail exchanger= -mail.python.org.Authoritative answers can be found from: mail.python.org Internet address=82.94.164.166mail.python.org has AAAA address2001:888: -:d:: A6exit code:0

Inter-process communication

ProcessThere is definitely a need for communication, and the operating system provides many mechanisms for communicating between processes. The Python multiprocessing module wraps the underlying mechanism, providing, and Queue Pipes so on, a variety of ways to exchange data.

QueueFor example, we create two sub-processes in the parent process, one to Queue write the data, and one to Queue read the data from the inside:

 frommultiprocessing Import process, Queueimport OS, time, random# code that writes the data processing execution: def write (q): Print ('Process to write:%s'%os.getpid ()) forValueinch['A','B','C']: Print ('Put%s to queue ...'%value) q.put (value) time.sleep (Random.random ()) # Read data Process Execution code: def read (q): Print ('Process to read:%s'%os.getpid ()) whileTrue:value= Q.Get(True) print ('Get%s from queue.'%value)if__name__=='__main__': # Parent process creates a queue and passes to each sub-process: Q=Queue () PW= Process (Target=write, args=(q,)) PR= Process (Target=read, args=(q,)) # Start child process PW, write: Pw.start () # Start child process PR, read: Pr.start () # Wait for PW to end: Pw.join () # PR process is a dead loop, can't wait for its end, only Can forcibly terminate: Pr.terminate ()

The results of the operation are as follows:

5056350564 from thefrom queue.

Under Unix/linux, the multiprocessing module encapsulates the fork() call so that we don't need to focus on fork() the details. Since Windows is not fork called, therefore, the multiprocessing need to "emulate" the fork effect, all Python objects of the parent process must be serialized through pickle and then passed to the child process, all, if multiprocessing the Windows downgrade fails, First consider whether the pickle failed.

Summary

Under Unix/linux, you can use fork() calls to implement multiple processes.

To implement multi-process across platforms, you can use multiprocessing modules.

Inter-process communication is Queue achieved through, Pipes and so on.

36 Multi-process