Notes for using multiprocessing in python: pythonprocessing

Source: Internet
Author: User

Notes for using multiprocessing in python: pythonprocessing

We often like to write code like this when writing a program.

import MySQLdbimport timefrom multiprocessing import Processconn = MySQLdb.connect('localhost', 'vearne', 'xx', 'test')def f(name):    for i in xrange(10):        cursor = conn.cursor()        sql = "insert into car(name) values(%s)"        param = [(name)]        print param        #time.sleep(1)        n = cursor.execute(sql,param)        cursor.close()        conn.commit()if __name__ == '__main__':    for i in xrange(10):        p = Process(target=f, args=('bob',))        p.start()

Is there a problem with the above program?
In the case of a single process, the above program should be normal, but in the case of multiple processes, it is wrong.

First look at the following source code

Class Process (object): '''process objects represent activity that is run in a separate Process The class is analagous to 'threading. thread ''' _ Popen = None def _ init _ (self, group = None, target = None, name = None, args = (), kwargs = {}): assert group is None, 'Group argument must be None for now 'count = _ current_process. _ counter. next () self. _ identity = _ current_process. _ identity + (count,) self. _ authkey = _ current_process. _ authkey self. _ daemonic = _ current_process. _ daemonic self. _ tempdir = _ current_process. _ tempdir self. _ parent_pid = OS. getpid () self. _ popen = None self. _ target = target self. _ args = tuple (args) self. _ kwargs = dict (kwargs) self. _ name = name or type (self ). _ name _ + '-' + \':'. join (str (I) for I in self. _ identity) def run (self): ''' Method to be run in sub-process; can be overridden in sub-class ''' if self. _ target: self. _ target (* self. _ args, ** self. _ kwargs) def start (self): '''start child process''' assert self. _ popen is None, 'Could not start a process twice 'assert self. _ parent_pid = OS. getpid (), \ 'Can only start a process object created by current process' assert not _ current_process. _ daemonic, \ 'daemonic processes are not allowed to have children '_ cleanup () if self. _ Popen is not None: Popen = self. _ Popen else: from. forking import Popen self. _ popen = Popen (self) # -- create a Popen object -- _ current_process. _ children. add (self) # omitting some code ...... def _ bootstrap (self): # -- _ bootstrap function -- from. import util global _ current_process try: self. _ children = set () self. _ counter = itertools. count (1) try: sys. stdin. close () sys. stdin = open (OS. devnull) doesn t (OSError, ValueError): pass _ current_process = self util. _ finalizer_registry.clear () util. _ run_after_forkers () util.info ('child process calling self. run () ') try: self. run () # -- call the run function -- exitcode = 0 finally: util. _ exit_function () doesn t SystemExit, e: if not e. args: exitcode = 1 elif isinstance (e. args [0], int): exitcode = e. args [0] else: sys. stderr. write (str (e. args [0]) + '\ n') sys. stderr. flush () exitcode = 0 if isinstance (e. args [0], str) else 1 response T: exitcode = 1 import traceback sys. stderr. write ('process % s: \ n' % self. name) sys. stderr. flush () traceback. print_exc () util.info ('process exiting with exitcode % d' % exitcode) return exitcode

From. forking import Popen Definition

Class Popen (object): def _ init _ (self, process_obj): sys. stdout. flush () sys. stderr. flush () self. returncode = None self. pid = OS. fork () # -- fork sub-process -- # The fork function is called once and will be returned twice, once in the parent process, the returned pid value is greater than 0 # a return is made in the sub-process, and the returned pid value is equal to 0 if self. pid = 0: # if the pid value is equal to 0, the following code is executed in the sub-process if 'rand' in sys. modules: import random. seed () code = process_obj. _ bootstrap () # -- call _ bootstrap function -- sys. stdout. flush () sys. stderr. flush () OS. _ exit (code)

From the code, we can see that python's multiprocessing uses fork to create a sub-process and executes the run function in the sub-process.

man fork 

The following information is displayed:

Fork() causes creation of a new process.  The new process (child process) is an exact copy of the calling process (parent process) except for the following:           o   The child process has a unique process ID.           o   The child process has a different parent process ID (i.e., the process ID of the parent process).           o   The child process has its own copy of the parent's descriptors.  These descriptors reference the same underlying objects, so that, for instance, file pointers in file objects are shared between the child and the parent, so that an lseek(2) on a descriptor in the child process can affect a subsequent read or write by the parent.  This descriptor copying is also used by the shell to establish standard input and output for newly created processes as well as to set up pipes.           o   The child processes resource utilizations are set to 0; see setrlimit(2).

The child process created by the fork function is a full copy of the parent process and has the same file descriptor. In this way, if a connection is created in the parent process, an unexpected error occurs when the parent process shares a connection with multiple child processes.
(Each connection has an independent read buffer and write buffer. Operations by multiple processes on the read buffer and write buffer will cause data confusion)

Therefore, we should create a connection in the sub-process to avoid problems.

Import MySQLdbimport timefrom multiprocessing import Processclass SLWorker (Process): def _ init _ (self): super (SLWorker, self ). _ init _ () self. conn = None def run (self): # *** Note: ** the connection delays loading. That is to say, the connection is created if self in the sub-process. conn = None: self. conn = MySQLdb. connect ('localhost', 'vearne ', 'xxx', 'test') for I in xrange (10): cursor = self. conn. cursor () SQL = "insert into car (name) values (% s)" name = "bob" pa Ram = [(name)] print param # time. sleep (30) n = cursor.exe cute (SQL, param) cursor. close () self. conn. commit () def _ del _ (self): if self. conn! = None: self. conn. close () if _ name _ = '_ main _': ll = [] for I in xrange (10): p = SLWorker () p. start () ll. append (p) for p in ll: p. join ()

The answer is simplyCreate a connection in a sub-process, OrConnection delay CreationThis problem can be solved.
In fact, there are many connection pools that are delayed in creating connections. If you have not carefully looked at them, I will share some research with you.

PS:Celery and rq both have such problems. Please pay enough attention to them.

Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.