The process of Python

To get the Python program to implement multi-process (multiprocessing), we first understand the operating system knowledge.

The Unix/linux operating system provides a fork() system call, which is very special. A normal function call, called once, is returned once, but fork() called once, and returned two times, because the operating system automatically replicates the current process (called the parent process) with a copy (called a child process), and then returns within the parent and child processes, respectively.

The child process returns forever 0 , and the parent process returns the ID of the child process. The reason for this is that a parent process can fork out many child processes, so the parent process has to note the ID of each child process, and the child process only needs to invoke getppid() the ID of the parent process.

Python's os modules encapsulate common system calls, including fork the ability to easily create child processes in a Python program:

import osprint(‘Process (%s) start...‘ % os.getpid())# Only works on Unix/Linux/Mac:pid = os.fork()if pid == 0: print(‘I am child process (%s) and my parent is %s.‘ % (os.getpid(), os.getppid()))else: print(‘I (%s) just created a child process (%s).‘ % (os.getpid(), pid))

The results of the operation are as follows:

start...I (876) just created a child process (877).I am child process (877) and my parent is 876.

forkThe above code cannot be run on Windows because Windows is not called. Because the Mac system is based on the BSD (Unix) kernel, so, running under the Mac is no problem, we recommend that you learn python! with Mac

With a fork call, a process can replicate a child process to handle a new task when it receives a new task, and the common Apache server is the parent process listening on the port, and whenever there is a new HTTP request, fork out the child process to process the new HTTP request.

Multiprocessing

If you are going to write a multi-process service program, Unix/linux is undoubtedly the right choice. Because Windows didn't fork call, wouldn't it be possible to write multi-process programs in Python on Windows?

Because Python is cross-platform, nature should also provide a cross-platform, multi-process support. multiprocessingmodules are multi-process modules with cross-platform versions.

multiprocessingThe module provides a Process class to represent a process object, and the following example demonstrates starting a child process and waiting for it to end:

From multiprocessing import Process
Import OS
Import time
Import Random

# code to be executed by the child process

def run_childprocess (name):
    Print (Time.ctime ())
    Print ('%s (id:%s) Purchase success '% (Name,os.getpid ()))

if __name__ = = ' __main__ ':
    Print (' mainprocess, parent process%s '% os.getpid ())
    p = Process (target=run_childprocess,args= (' Dava ',))
    P2 = Process (target=run_childprocess,args= (' Diamond Baby ',))
    P3 = Process (target=run_childprocess,args= (' Water Doll ',))
    P4 = Process (target=run_childprocess,args= (' Fire Eva '))
    Print (' All children process would start. ')
    P.start ()
    P2.start ()
    P3.start ()
    P4.start ()
    P.join ()
    Print (' child process end. ')
    #如果要启动大量的进程, the process pool should be used

Pool

If you want to start a large number of child processes, you can create the child processes in batches using the process pool:

From multiprocessing import Pool
Import OS
Import time
Import Random

def run_childprocess (name):
    Print (Time.ctime ())
    Print ('%s (id:%s) Purchase success '% (Name,os.getpid ()))


if __name__ = = ' __main__ ':
    Print (' main main process '%s. '% Os.getpid ())
    p = Pool (5)
    NameList = [' Dava ', ' King Kong ', ' Fire Eva ', ' Water Doll ', ' Stealth doll ', ' naïve baby ']
    For I in range (6):
        P.apply_async (run_childprocess,args= (Namelist[i],))
    Print (' Waiting for all subprocesses done ... ')
    P.close ()
    P.join ()
    Print (' All subprocesses do. ')

Invoking a method on an Pool object join() waits for all child processes to complete before the call must be called before the call join() close() close() can continue to add new Process .

Child process

Many times, a child process is not itself, but an external process. After we have created the child process, we also need to control the input and output of the child process.

subprocessThe module allows us to start a subprocess very conveniently and then control its input and output.

The following example shows how to run a command in Python code nslookup www.python.org , which works just like the command line:

import subprocessprint(‘$ nslookup www.python.org‘)r = subprocess.call([‘nslookup‘, ‘www.python.org‘])print(‘Exit code:‘, r)

Operation Result:

$ nslookup www.python.orgServer:        192.168.19.4Address: 192.168.19.4#53Non-authoritative answer:www.python.org canonical name = python.map.fastly.net.Name: python.map.fastly.netAddress: 199.27.79.223Exit code: 0

Child process

Many times, a child process is not itself, but an external process. After we have created the child process, we also need to control the input and output of the child process.

subprocessThe module allows us to start a subprocess very conveniently and then control its input and output.

The following example shows how to run a command in Python code nslookup www.python.org , which works just like the command line:

import subprocessprint(‘$ nslookup www.python.org‘)r = subprocess.call([‘nslookup‘, ‘www.python.org‘])print(‘Exit code:‘, r)

Operation Result:

$ nslookup www.python.orgServer:        192.168.19.4Address: 192.168.19.4#53Non-authoritative answer:www.python.org canonical name = python.map.fastly.net.Name: python.map.fastly.netAddress: 199.27.79.223Exit code: 0

If the child process also needs input, it can be communicate() entered by method:

import subprocessprint(‘$ nslookup‘)p = subprocess.Popen([‘nslookup‘], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)output, err = p.communicate(b‘set q=mx\npython.org\nexit\n‘)print(output.decode(‘utf-8‘))print(‘Exit code:‘, p.returncode)

The above code is equivalent to executing the command at the command line and nslookup then manually entering:

set q=mxpython.orgexit

The results of the operation are as follows:

$ nslookupServer:192.168.19.4address: Span class= "number" >192.168. 19.4 #53 non-authoritative answer:python.org mail exchanger = 50 Mail.python.org. Authoritative answers can be found from:mail.python.org Internet address =  82.94.164.166mail.python.org has AAAA address  2001:888:2000 :d ::a6exit code: 0           

Inter-process communication

ProcessThere is definitely a need for communication, and the operating system provides many mechanisms for communicating between processes. The Python multiprocessing module wraps the underlying mechanism, providing, and Queue Pipes so on, a variety of ways to exchange data.

QueueFor example, we create two sub-processes in the parent process, one to Queue write the data, and one to Queue read the data from the inside:

From multiprocessingImport Process, QueueImport OS, time, random# Write the code for the data Process execution:DefWrite(q): Print (' Process to write:%s '% os.getpid ())For valuein [' A ',' B ',' C ']: print (' Put%s to queue ... '% value ' q.put (value) time.sleep (Random.random ())# Read Data Process Execution code:def read(q): Print (' Process to read:%s '% os.getpid ()) and true:value = Q.get (True) print (' get%s from queue. ') % value)if __name__==' __main__ ': # Parent process creates a queue and passes to each sub-process: q = queue () PW = Process (Target=write, args= (q,)) PR = Process (Target=read, args= (Q,)) # start subprocess pw, write: Pw.start () # start child process PR, read: Pr.start () # Wait for PW to end: Pw.join () # PR process is a dead loop, can not wait for its end, can only forcibly terminate: Pr.terminate ()             

The results of the operation are as follows:

50563Put A to queue...Process to read: 50564Get A from queue.Put B to queue...Get B from queue.Put C to queue...Get C from queue.

Under Unix/linux, the multiprocessing module encapsulates the fork() call so that we don't need to focus on fork() the details. Since Windows is not fork called, therefore, the multiprocessing need to "emulate" the fork effect, all Python objects of the parent process must be serialized through pickle and then passed to the child process, all, if multiprocessing the Windows downgrade fails, First consider whether the pickle failed.

The process of Python