Preliminary analysis of multi-process programming in Python-Python tutorial

This article mainly introduces the preliminary analysis of multi-process programming in Python. using multi-process programming has always been a key and difficult point in Python programming, for more information about how to implement multiprocessing in a Python program, see.

The Unix/Linux operating system provides a fork () system call, which is very special. A common function is called once and returns once, but fork () is called once and returns twice because the operating system automatically calls the current process (called the parent process) copy a copy (called a child process), and then return the results in the parent process and child process respectively.

The child process always returns 0, and the parent process returns the child process ID. The reason for doing so is that a parent process can fork many sub-processes, so the parent process needs to write down the ID of each sub-process, and the sub-process only needs to call getppid () you can get the ID of the parent process.

The Python OS module encapsulates common system calls, including fork. you can easily create sub-processes in Python programs:

# multiprocessing.pyimport osprint 'Process (%s) start...' % os.getpid()pid = os.fork()if pid==0:  print 'I am child process (%s) and my parent is %s.' % (os.getpid(), os.getppid())else:  print 'I (%s) just created a child process (%s).' % (os.getpid(), pid)

The running result is as follows:

Process (876) start...I (876) just created a child process (877).I am child process (877) and my parent is 876.

Because Windows does not call fork, the above code cannot run on Windows. Because the Mac system is based on the BSD (Unix) kernel, it is no problem to run it on Mac. we recommend that you use Mac to learn Python!

With the fork call, a process can copy a sub-process to handle a new task when it receives a new task. the common Apache server is listening to the port by the parent process, every time there is a new http request, fork sends a sub-process to handle the new http request.
Multiprocessing

If you plan to write multi-process service programs, Unix/Linux is undoubtedly the right choice. Since Windows does not have a fork call, is it impossible to write multi-process programs in Python on Windows?

Because Python is cross-platform, it should also provide a cross-platform multi-process support. The multiprocessing module is a multi-process module of the cross-platform version.

The multiprocessing module provides a Process class to represent a Process object. the following example shows how to start a sub-Process and wait for it to end:

From multiprocessing import Processimport OS # code to be executed by the sub-process def run_proc (name): print 'run child process % s (% s )... '% (name, OS. getpid () if _ name __= = '_ main _': print 'parent process % s. '% OS. getpid () p = Process (target = run_proc, args = ('test',) print 'process will start. 'P. start () p. join () print 'process end.'

The execution result is as follows:

Parent process 928.Process will start.Run child process test (929)...Process end.

When creating a sub-Process, you only need to input a parameter for executing the function and function, create a Process instance, and start it using the start () method. This makes Process creation easier than fork.

The join () method can continue to run after the sub-process ends. it is usually used for synchronization between processes.
Pool

If you want to start a large number of sub-processes, you can use the process pool to create sub-processes in batches:

from multiprocessing import Poolimport os, time, randomdef long_time_task(name):  print 'Run task %s (%s)...' % (name, os.getpid())  start = time.time()  time.sleep(random.random() * 3)  end = time.time()  print 'Task %s runs %0.2f seconds.' % (name, (end - start))if __name__=='__main__':  print 'Parent process %s.' % os.getpid()  p = Pool()  for i in range(5):    p.apply_async(long_time_task, args=(i,))  print 'Waiting for all subprocesses done...'  p.close()  p.join()  print 'All subprocesses done.'

The execution result is as follows:

Parent process 669.Waiting for all subprocesses done...Run task 0 (671)...Run task 1 (672)...Run task 2 (673)...Run task 3 (674)...Task 2 runs 0.14 seconds.Run task 4 (673)...Task 1 runs 0.27 seconds.Task 3 runs 0.86 seconds.Task 0 runs 1.41 seconds.Task 4 runs 1.91 seconds.All subprocesses done.

Code explanation:

To call the join () method for a Pool object, wait until all sub-processes are executed. before calling join (), you must call close () and close () then you cannot add a new Process.

Note that the output results of task 0, 1, and 2 are executed immediately, and task 4 is executed only after a previous task is completed, this is because the default size of the Pool is 4 on my computer, so a maximum of four processes can be executed simultaneously. This is a deliberate design restriction of the Pool, not an operating system restriction. If Changed:

p = Pool(5)

You can run five processes at the same time.

Because the default size of the Pool is the number of CPU cores, if you unfortunately have 8-core CPU, you need to submit at least 9 sub-processes to see the above wait effect.
Inter-process communication

Processes must communicate with each other. the operating system provides many mechanisms for inter-Process communication. The Python multiprocessing module encapsulates underlying mechanisms and provides multiple methods such as Queue and Pipes for data exchange.

Taking Queue as an example, we create two sub-processes in the parent process, one is writing data to the Queue, and the other is reading data from the Queue:

From multiprocessing import Process, Queueimport OS, time, random # write data Process execution code: def write (q): for value in ['A', 'B ', 'C']: print 'put % s to queue... '% value q. put (value) time. sleep (random. random () # code executed by the read data process: def read (q): while True: value = q. get (True) print 'get % s from queue. '% valueif _ name __= =' _ main _ ': # the parent process creates a Queue and sends it to each sub-process: q = Queue () pw = Process (target = write, args = (q,) pr = Process (target = read, args = (q,) # promoter Process pw, write: pw. start () # promoter process pr, read: pr. start () # wait for the completion of pw: pw. join () # In the pr process is an endless loop and cannot wait until it ends. it can only be forcibly terminated: pr. terminate ()

The running result is as follows:

Put A to queue...Get A from queue.Put B to queue...Get B from queue.Put C to queue...Get C from queue.

In Unix/Linux, the multiprocessing module encapsulates fork () calls, so that we do not need to pay attention to the details of fork. Because Windows does not have a fork call, therefore, multiprocessing needs to "simulate" the fork effect. All Python objects of the parent process must be serialized by pickle and then transmitted to the child process, if multiprocessing fails to be called in Windows, consider whether the pickle fails.
Summary

In Unix/Linux, you can use fork () to call multiple processes.

To implement cross-platform multi-process, you can use the multiprocessing module.

Inter-process communication is implemented through Queue and Pipes.