Python multi-process Programming

Source: Internet
Author: User

Python is a simple and effective method for implementing program programming. However, one drawback of this method is that it does not always increase the speed of the application, because the global interpreter lock (Gil) effectively limits the thread to a single core. If you need to use all the cores in the computer, you usually need to use the fork operation to increase the speed. It is difficult to process a process group because all calls need to be coordinated for communication between processes, which usually makes things more complicated.

Since version 2.6, Python includes a module named "multiprocessing" to help process. The API of the Process Module is similar to that of the thread API, but there are also some differences that need special attention. One of the main differences is some subtle underlying behaviors of processes, which are never completely abstracted by advanced APIs.


Introduction to OS. Fork

There are some obvious differences in the working principles of concurrency between processes and threads. When the fork process is executed, the operating system will create a new sub-process with an ID and copy the status of the parent process (memory, environment variables, etc ). First, let's take a look at a very basic fork operation in Python before we actually use the process module.

#!/usr/bin/env python
"""fork example"""
import os

def main():
child_pid = os.fork()
if child_pid == 0:
print "Child Process: PID# %s" % os.getpid()
else:
print "Parent Process: PID# %s" % os.getpid()

if __name__ == "__main__":
main()

Next let's take a look at the system variable transfer between processes.

#!/usr/bin/env python
"""A fork that demonstrates a copied environment"""

import os

def main():
os.environ['prefix']="zip"
print "prefix environmental variable set to: %s" % os.environ['prefix']
os.environ['prefix']="tar"
print "prefixenvironmental variable changed to: %s" % os.environ['prefix']
child_pid = os.fork()
if child_pid == 0:
print "Child Process: PID# %s" % os.getpid()
print "Child prefixenvironmental variable == %s" % os.environ['prefix']
else:
print "Parent Process: PID# %s" % os.getpid()
print "Parent prefix environmental variable == %s" % osenviron['prefix']

if __name__ == "__main__":
main()

The output of fork is given below:

prefix environmental variable set to: zip
prefix environmental variable changed to: tar
Parent Process: PID# 52333
Parent prefix environmental variable == tar
Child Process: PID# 52334
Child prefix environmental variable == tar

In the output, we can see that the "modified" environment variable prefix is left in the child process and parent process. You can further test this example by modifying the environment variable in the parent process. You can see that the child process is completely independent and has its own life.

Note that the sub-process module can also be used for Fork processes, but the implementation method is not as complex as the multi-process module.

The following is a practical example of multi-process processing.

#!/usr/bin/env python
from multiprocessing import Process
import os
import time

def sleeper(name, seconds):
print 'starting child process with id: ', os.getpid()
print 'parent process:', os.getppid()
print 'sleeping for %s ' % seconds
time.sleep(seconds)
print "Done sleeping"

if __name__ == '__main__':
print "in parent process (id %s)" % os.getpid()
p = Process(target=sleeper, args=('bob', 5))
p.start()
print "in parent process after child process start"
print "parent process about to join child process"
p.join()
print "in parent process after child process join"
print "parent process exiting with id ", os.getpid()
print "The parent's parent process:", os.getppid()

The output is as follows:

in parent process (id 5245)
in parent process after child process start
parent process about to join child process
starting child process with id: 5246
parent process: 5245
sleeping for 5
Done sleeping
in parent process after child process join
parent process exiting with id 5245
The parent's parent process: 5231

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.