Python Learning notes Multi-process

To get the Python program to implement multiple processes, let's start with the knowledge of the operating system

Unix/linux operating system provides a fork () system call, he is very special, ordinary function call, call once, return once, but fork call once,

Returns two times because the operating system automatically copies the current process, called the parent process, and then returns it in the parent and child processes, respectively.

The child process always returns 0, and the parent process returns the ID of the child process. The reason for this is that a parent process can fork out many child processes, so the parent process has to be written down

The ID of the child process, and the child process only needs to call Getppid () to get the ID of the parent process

Python's OS module encapsulates common system calls, including fork, which makes it easy to create sub-processes in a Python program

ImportOSPrint('Process (%s) Start ...'%os.getpid ())#Only works on Unix/linux/mac:PID =os.fork ()ifPID = =0:Print('I AM Child process (%s) and my parent is%s.'%(Os.getpid (), Os.getppid ()))Else:    Print('I (%s) just created a child process (%s).'% (Os.getpid (), PID))

With a fork call, a process can replicate a child process to handle a new task when it receives a new task, and the common Apache server is the parent process listening on the port, and whenever there is a new HTTP request, fork out the child process to process the new HTTP request.

forkThe above code cannot be run on Windows because Windows is not called. Because the Mac system is based on the BSD (Unix) kernel, so, running under the Mac is no problem, we recommend that you learn python! with Mac

With a fork call, a process can replicate a child process to handle a new task when it receives a new task, and the common Apache server is the parent process listening on the port, and whenever there is a new HTTP request, fork out the child process to process the new HTTP request.

Multiprocessing

If you are going to write a multi-process service program, Unix/linux is undoubtedly the right choice. Because Windows didn't fork call, wouldn't it be possible to write multi-process programs in Python on Windows?

Because Python is cross-platform, nature should also provide a cross-platform, multi-process support. multiprocessingmodules are multi-process modules with cross-platform versions.

multiprocessingThe module provides a Process class to represent a process object, and the following example demonstrates starting a child process and waiting for it to end:

 fromMultiprocessingImportProcessImportOS#code to be executed by the child processdefRun_proc (name):Print('Run Child process%s (%s) ...'%(name, Os.getpid ()))if __name__=='__main__':    Print('Parent process%s.'%os.getpid ()) P= Process (Target=run_proc, args= ('Test',)) Print('Child process would start.') P.start () P.join ()Print('Child process end.')

When you create a child process, you only need to pass in a parameter that executes functions and functions, create a process instance, and start with the start () method, so that the creation process is simpler than fork (). The join () method can wait for the child process to end before continuing to run, typically for inter-process synchronization.

Pool

If you want to start a large number of child processes, you can create the child processes in batches using the process pool:

 fromMultiprocessingImportPoolImportOS, time, RandomdefLong_time_task (name):Print('Run Task%s (%s) ...'%(name, Os.getpid ())) Start=time.time () time.sleep (Random.random ()* 3) End=time.time ()Print('Task%s runs%0.2f seconds.'% (name, (End-start )))if __name__=='__main__':    Print('Parent process%s.'%os.getpid ()) P= Pool (4)     forIinchRange (5): P.apply_async (long_time_task, args=(i,))Print('waiting-subprocesses done ...') P.close () P.join ()Print('All subprocesses is done .')

The execution results are similar to the following

Parent process 669 for all subprocesses done ... Run Task 0 (6711 (6722 (6733 (6742 runs 0.144 (6731 runs 0.27< c9>3 runs 0.861.414 runs 1.91 seconds. All subprocesses is done.

Child process

Many times, a child process is not itself, but an external process. After we have created the child process, we also need to control the input and output of the child process.

subprocessThe module allows us to start a subprocess very conveniently and then control its input and output.

The following example shows how to run a command in Python code nslookup www.python.org , which works just like the command line:

Inter-process communication

ProcessThere is definitely a need for communication, and the operating system provides many mechanisms for communicating between processes. The Python multiprocessing module wraps the underlying mechanism, providing, and Queue Pipes so on, a variety of ways to exchange data.

QueueFor example, we create two sub-processes in the parent process, one to Queue write the data, and one to Queue read the data from the inside:

 fromMultiprocessingImportProcess, QueueImportOS, time, Random#code to write the data Process execution:defWrite (q):Print('Process to write:%s'%os.getpid ()) forValueinch['A','B','C']:        Print('Put%s to queue ...'%value) q.put (value) time.sleep (Random.random ())#read the code that the data process executes:defRead (q):Print('Process to read:%s'%os.getpid ()) whileTrue:value=q.get (True)Print('Get%s from queue.'%value)if __name__=='__main__':    #The parent process creates a queue and passes it to each child process:Q =Queue () PW= Process (Target=write, args=(q,)) PR= Process (Target=read, args=(q,))#Start child process PW, write:Pw.start ()#start child process PR, read:Pr.start ()#wait for PW to end:Pw.join ()#PR process is a dead loop, can not wait for its end, can only forcibly terminate:Pr.terminate ()

Under Unix/linux, the multiprocessing module encapsulates the fork() call so that we don't need to focus on fork() the details. Since Windows is not fork called, therefore, the multiprocessing need to "emulate" the fork effect, all Python objects of the parent process must be serialized through pickle and then passed to the child process, all, if multiprocessing the Windows downgrade fails, First consider whether the pickle failed.

Python Learning notes Multi-process