Python multi-process and python Process

Python multi-process and python Process

Python3.6

Multi-Process

Multi-Process Table of Contents

• 1. Multi-Process
  • 1.1 comparison of linux/unix/win startup Methods
    • 1.1.1. Default spawn win method, supporting unix/win
    • 1.1.2. Default fork unix method, only supports unix
    • 1.1.3. forkserver this method is available when the platform supports unix pipeline files
    • 1.1.4. promoter Process
    • 1.1.5. Select the start Method
    • 1.1.6. Other content
  • 1.2. Process Communication
    • 1.2.1. Queue
    • 1.2.2. TODO pipeline Pipes
    • 1.2.3. Instance
  • 1.3. Process Synchronization
    • 1.3.1. Lock
    • 1.3.2. Instance
  • 1.4. Process Pool
    • 1.4.1. Create a Pool ([processes [, initalizer [, initargs [, maxtasksperchild [, context])
    • 1.4.2. AsyncResult applyasync, returned objects of mapasync and starmapasync
    • 1.4.3. Instance
  • 1.5. Resource Sharing
    • 1.5.1. Use Array and Value as the bucket to save resources to be shared
    • 1.5.2. Instance
    • 1.5.3. Use the service process server process

1 multi-process 1.1 linux/unix/win Startup Mode Comparison

The multi-process mechanism used by the system on different platforms is different, so there are three different ways to enable multi-process in python implementation.

1.1.1 default spawn win method, supporting unix/win

The parent process starts a new python interpreter sub-process. Only the file descriptor and handle of the parent process that runs the run () method are not inherited. The speed of this method is the slowest among the three methods.

1. Run () This method is the target parameter, which is a callable object.
   

   After testing, when run calls a function, the resource still cannot be retained. Only one callable object is called and the initialization method (init) has other parameters except self.

   Def hello (): passclass Hello (): def _ init _ (self, name): print (name) hello. _ init _ () # The parameter is not required, so the resource cannot be retained. _ init _ ('name') # resources can be retained only when parameters are passed

   Instance:

   From multiprocessing import Process, Queueimport multiprocessing as mpimport osdef hello_procs (name): print ("this is the {} Process ". format (name, OS. getpid () print ("parent process ID % s" % OS. getppid () try: print (sourceA) print ("Call successful, because the spawn method is not used") failed t Exception as e: print ("Call failed, because this resource is not a required resource ") print (e) class Sou (): def _ init _ (self, soua): self. soua = soua def show_soua (self): print (self. soua) def _ call _ (self): self. show_soua () if _ name _ = '_ main __': sourceA = "AA" sourceB = "BB" # Use forkserver to perform consistent performance with spawn in terms of resource inheritance. set_start_method ('spawn ') print ("the current process id is % s" % OS. getpid () # p = Process (target = hello_procs, args = ('first process',) a = Sou (sourceA) p = Process (target =) "above is the transfer method without parameters. To pass parameters to callable functions, you need to change it to _ call _ (self, parameter) next, modify the Process parameter to (target = a, args = ('parameter',) "print ('start into subprocess') p. start () p. join () print ("process ended ")

1.1.2 default fork unix method, only supports unix

Using unix fork () [OS. fork ()] to create a sub-process of the current interpreter to obtain all the resources of the parent process is difficult to control.

1.1.3 forkserver this method is available when the platform supports unix pipeline files

When using this method, when a multi-process is enabled, an additional service process is started. When a sub-process is required, the parent process requests the service process and obtains a sub-process. Because the service process is single-threaded, therefore, this method is thread-safe.

1.1.4 promoter Process

1. Creating a process object indicates the activity that runs in a separate process
   

   Process (group = None, target = None, name = None, args = (), kwargs = {}, *, daemon = None) the group is always None, only to match threading. thread-compatible target is the object to be called by run (). It must be an executable name with no intention, the location parameter of the name args callable object kwargs callable object's keyword parameter daemon True/False/None is related to process inheritance

2. Run () indicates the method of process activity
3. Start () start Process Activity
4. Isalive () checks whether sub-processes are alive. Only sub-processes can be detected.
5. Join ([timeout]) blocks the process that calls this method
   

   None blocking, until the process is completed a positive number blocking timeout seconds

6. Daemon identifies whether the process is a daemon. True is True. False is not. None is inherited from the upper layer.
   

   The main process is not a daemon, so as long as it is not explicitly specified as True, all created processes are not daemon.

   From multiprocessing import Process as psimport sysdef hello (*, a = None): if a = None: pass else: print (a) def not_sh (): global pish, pfsh, ex print ("non-daemon % s" % ex) pish = ps (target = is_sh, args = (1,), daemon = True) pfsh = ps (target = is_sh, args = ('non-daem',) # daemon inherits the attribute pish from the upper layer for None. start () pfsh. start () for I in range (1): # When this value is 1, you can see that the daemon exits print ("second main process") before printing the message ") ex = "changed" def is_sh (jc_type): if jc_type = 1: prin T ("daemon") for I in range (1000): print ("daemon") else: print () print ("not daemon ") for I in range (100): print ("non-daemon") if _ name _ = '_ main _': p = ps (target = hello, name = 'hello', args = (), kwargs = {'A': 'A'}) print ("whether the process is alive % s" % p. is_alive () p. start () print ("daemon? % S "% p. daemon) print ("whether the process is alive % s" % p. is_alive () # create a non-daemon pish, pfsh = None, None ex = "original" pnsh = ps (target = not_sh) pnsh. start () pnsh. join () print (ex)

1.1.5 select the start Method

Import multiprocessingmultiprocessing. set_start_method ('spawn ') # Name of the Input Method # This method can be used at most once in a program

1.1.6 other content

1. You can use global to send data to sub-processes. However, modifications made to the data by sub-processes are not reported to the parent process.

1.2 Process Communication 1.2.1 Queue

1. Create a Queue ([maxsize]) and set the maximum value.
2. Qsize () returns the approximate size of the queue (inaccurate because of concurrency)NotImplementedError is thrown back on MAC.
3. Whether empty () is empty, inaccurate
4. Full () queue is full, inaccurate
5. Put (obj [, block [, timeout])
6. Putnowait (obj) is equivalent to put (obj, False)
7. Get ()
8. Getnowite ()
9. Close () has no return value. If the queue is closed, the background thread refreshes the data to the MPs queue once. After the data is closed, the Operation will throw an exception OSError: handle is closed.
10. Jointhread () can only be called after close (). It is blocked until the background thread exits and ensures that the data is refreshed to the MPs queue.
11. Canceljointhread () immediately closes the queue and does not wait for the background thread to refresh the data to the pipeline

1.2.2 example of the TODO pipeline Pipes1.2.3

"Inter-Process communication" from multiprocessing import Process, Queue, Pipeimport OS, randomdef write (m): print ('process: % s' % OS. getpid () m. put ('data A') "put (obj [, block [, timeout]) put the value in the queue when the block is True (default ), if the value of timeout is None (default value), no exception will be thrown. It will wait until the value is added to the queue. When the value of timeout is a positive value, wait for timeout seconds, and the queue will be thrown when the timeout is reached. full exception: When the block is False, an exception is thrown immediately when the queue cannot be entered "def read (m): print ('process: % s' % OS. getpid () try: print (m. qsize () failed t Exception as e: print ("Exception thrown on MAC") finally: value = m. get () "" get ([block [, timeout]) get a value and delete it. When the block is True (default) and the timeout is None (default ), if the value of timeout is positive only when there is content in the queue, the queue will be blocked when there is no value in the queue, and then the queue will be thrown if there is no value. if the Empty exception block is False, an exception is thrown immediately when no value is set. "print (value) if _ name _ = '_ main _': q = Queue () pw = Process (target = write, name = 'write process', args = (q,) print ('start to write data % s' % pw. name, end = ':') pw. start () pr = Process (target = read, name = 'read process', args = (q,) print ('start to read data % s' % pr. name, end = ':') pr. start () # TODO Pipe transmits messages through pipelines

1.3 Process Synchronization 1.3.1 Lock

Once the thread gets the lock, any subsequent process will be blocked when the thread gets the lock.

1. Acquire (block = True, timeout = None)
   

   When the block is True, the method call will be blocked until the block is unlocked. When the block is True, the timeout value is a positive number. Therefore, a maximum of timeout seconds can be blocked. When the timeout value is negative, the blocking duration is 0, always blocked when None

2. Release () release lock
   

   A lock can be released by any object and may not be unlocked by a locked object.

1.3.2 instance

From multiprocessing import Process, Lockdef show_lock (l): # l. release () locks in try_get_lock can be unlocked here. acquire (True,-1) # The timeout length is a negative number. Even if it is locked, print ("normal function execution") def try_get_lock (l): l. acquire () print ("obtained lock") # l. release () if _ name _ = '_ main _': l = Lock () # l. acquire (True) pg = Process (target = try_get_lock, args = (l,) pg. start () ps = Process (target = show_lock, args = (l,) ps. start ()

1.4 process Pool 1.4.1 create a Pool ([processes [, initalizer [, initargs [, maxtasksperchild [, context])

Number of processes initializer if the value is not None, the context of the maxtasksperchild context working process called initializer (* initargs) when each worker starts.

This class implements context management

1. Apply (func [, args [, kwds])
   

   Use args and kwds to call func until the result is complete.

2. Applyasync (func [, args [, kwds [, callback [, errorcallback])
   

   Returns a result object. The returned object isAsyncResultWhen a callback (a callback object that can be called to accept a single parameter) is specified, callback is called when the call is completed. If the call fails, the call to the errorcallback callback should be completed immediately; otherwise, the thread will be blocked.

3. Map (func, iterable [, chunksize])
   

   Similar to the built-in Function map (), it blocks until the map is complete.

4. Mapasync (func, iterable [, chunksize [, callback [, errorcallback])
   

   Map () of the returned result ()

5. Imap (func, iterable [, chunkszie])
   

   The parameters of the inert map () chunkszie method are the same as those of the map () method.

6. Starmap (func, iterable [, chunksize]) iterable must be an iterative object
   

   Note:'Abc' is also an iteratable object. Once () is added, ('abc') cannot provide func. ('abc',) three parameters are passed to func instead of a whole.Correct practiceInput ('abc',),). Similarly, other iteratable content can be imported.

7. Starmapasync (func, iterable [, chuunksize [, callback [, errorback])
   

   Call func after splitting iterable and return a result object

8. Close ()
   

   Once the task is completed, exit the process

9. Terminate ()
   

   Stop the process immediately and exit

10. Join ()
    

    Wait until the process ends. Before that, you must call close or terminate.

1.4.2 AsyncResult applyasync, returned objects of mapasync and starmapasync

1. Get ([timeout]) returns the result and must arrive within timeout seconds. If the value of timeout is not None, the system will throw an exception TimeoutError if the value of timeout is not reached within N seconds.
2. Wait ([timeout]) waits for the result or waits for N seconds to time out
3. Ready () determines whether the returned result is ready.
4. Successful () returns whether the call is complete and no exception occurs.

1.4.3 instance

From multiprocessing import Pool, TimeoutError, Processimport timeimport osdef proc_pool (name): print ("asd") for I in range (5): print (str (I) + ': % s') # return "returned result value", "What if there are two? "Do not return more than one value, which leads to ambiguity in map calls (when map (func, [1, 2]) is used, [return value 1, return value 2] instead of [(return value 1, return Value 2), (return value 1, return value 2)]) When two values need to be returned explicitly, return a tuple # return ("returned result value", "second value ") return "return Value" def proc_err (name): raise Exceptiondef proc_mm (name): print ('this function has been called % s % s' % (name, type (name ))) return nameif _ name _ = '_ main _': print ("start thread Pool") p = Pool (4) for I in range (5 ): p. apply_async (proc_pool, args = ('cc',) # p. map (proc_pool, ['cc', 'dd']) def callback (name): print ("callback Function % s") def err_callback (err): try: print ("yc") failed t Exception as e: print ('exception occurred ') finally: print ("ww") mapr = p. map_async (proc_err, 'ee ', 3, callback, err_callback) # mapr. get () get the callable object return value # print ("returned result value % s" % mapr. get () # mmap = p. starmap (proc_mm, [('abc'), ('A')]) mmap = p. starmap (proc_mm, ('abc',),) mmaps = p. starmap_async (proc_mm, ('abc',),) list (mmap) print (mmaps. get () p. close () p. join ()

1.5 Resource Sharing 1.5.1 Use Array and Value as the storage space to save the resources to be shared

Value (typecodeortype, * args, lock = True) Array (typecodeortype, sizeorinitializer, *, lock = True)

1. What are the commonalities of Array and Value?
   

   When creating a bucket, when selecting the lock parameter, you can create a resource lock by default, but you can also choose to use an existing lock, when the lock is passed into an existing lock, it is affected by the lock. When it is set to False, the resource is not protected by the lock, resulting in thread security.

   Tyoecodeortype is the type code array used by the array module to represent an array of basic types, including: characters, integers, and floating-point numbers.

2. Differences
   

   Array stores a queue, Value stores a Value. The sizeofinitializer of the Array is the saved Array, and the length of the Array is also the length of the Array.

1.5.2 instance

"Process share content using Value, Array share content" from multiprocessing import Process, Value, Array, Lockdef f (n, a): n. value = 3.1415927 for I in range (len (a): a [I] =-a [I] def fun (l, strr): # Get the lock. When it is locked, wait for a maximum of 3 seconds to continue. acquire (True, 3) try: print (num. value) print (strr. value) print (chr (strr. value) failed t Exception as ex: print (ex) finally: print ('complete') if _ name _ = '_ main __': l = Lock () num = Value ('D', 0.0) arr = Array ('I', range (10) lisi =, 3] arrs = Array ('I', lisi) # Because python does not have the char type, you can only convert it to a number here, and then return strr = Value ('B ', ord ('C') p = Process (target = f, args = (num, arrs) pl = Process (target = fun, args = (l, strr )) # locking l. acquire () p. start () p. join () pl. start () pl. join () print (num. value) print (arrs [:])

1.5.3 use the service process server process

1. When Manager () is used, a management object is returned.
   

   This management object supports a wider range of types: list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Barrier, Queue, Value, Array

   This class implements context management

2. The two sub-classes of the Manager. The Manager () returns the SyncManager.
   
   1. BaseManager ([adress [, authkey])
      

      Adress is the address where the manager process listens for new links. If None is set to a random authkey, the authentication key is selected, and None is set to currentprocess (). authkey. Otherwise, the authkey must be the string currentprocess () to return the authentication key (byte string) of the current Process object authkey Process. When initializing multiprocessing, OS is used. urandom () assigns a random string to the master Process. When a Process object is created, it inherits the authentication key of its parent Process, but can be changed by setting the authkey as another byte string.

      1. Start ([initializer [, initargs])
         

         Promoter process to start the manager

      2. Getserver ()
         

         Returns the Server object, which indicates the actual Server under the manger control.

      3. Connect ()
         

         Link the local manager object to the remote Manager Process

      4. Shutdown ()
         

         Stop the manager Process, available only when start () is started

      5. Register (typeid [, callable [, proxytype [, exposed [, methodtotypeid [, createmethod])
         

         Register a type or callable class method with the Manager

         Typeid is the type identifier used to identify a specific type of shared object. It must be the callable string used to call the typeid type. proxytype is a subclass of BaseProxy, used to create a shared object proxy for typeid, None, the automatic creation of exposed is used to specify the method used by the proxy type. methodtotypeid: returns the public method of the proxy type. createmethod: determines whether to use the typeid creation method. The default value is True.

   2. SyncManager
      

      BaseManager is mainly used to create custom managers.

      1. Queue ([maxsize]) creates a queue. Queue object and returns its proxy
         

         This document describes how to use some Queue queues.

      2. Array (typecode, sequence) creates an Array and returns its proxy
         

         Demonstrate some Array usage in process Sharing

      3. Value (typecode, value) creates a Value and returns its proxy
         

         Demonstrate some Value usage in process Sharing

      4. Dict ([dict]) creates a dict and returns its proxy
      5. List ([list]) creates a list and returns its proxy
      6. Lock () creates a threading. Lock Object and returns its proxy
   3. Instance
      

      from multiprocessing import Process, Managerdef f(d, l, q, a, v, lo):    d[1] = '1'    d['2'] = 2    d[0.25] = None    q.put(100)    lo.acquire(True,3)    for i in range(len(a)):a[i]=1    v,value = 100    l.reverse()if __name__ == '__main__':    with Manager() as manager:d = manager.dict()l = manager.list(range(10))q = manager.Queue(10)a = manager.Array('i',[1,2,3])v = manager.Value('i',3)lo = manager.Lock()lo.acquire(True)p = Process(target=f, args=(d, l, q, a, v, lo))p.start()p.join()print(d)print(l)print("********")print(q.get())print(a[:])print(v.value)

Author: vz li Branch

Created: Sat

Emacs 25.1.1 (Org mode 8.2.10)

Validate