Python: threads, processes, and co---multiprocessing (4) module (1)

The multiprocessing module is a package provided by Python for multi-process development, and the multiprocessing package provides both local and remote concurrency, effectively avoiding global interpreter locks by using child processes rather than threads.

(i) Create Process class

Create the class of the process, its source code in the multiprocessing package of process.py, interested in the source of the understanding of the side to learn. It uses the same threading. The thread is about the same as it can be seen from its class definition, as follows:

Class process (object):     '     Process objects  represent activity that is run in a separate process     The class is analagous to  ' Threading. Thread '      '     _popen = none    def  __init__ (self, group=none, target=none, name=none, args= (),  kwargs={}):         assert group is None,  ' group argument  Must be none for now '         count = _ Current_process._counter.next ()         self._identity = _ current_process._identity +  (count,)         self._authkey  = _current_process._authkey        self._daemonic = _current_process._daemonic         self._tempdir = _current_process._tempdir         self._parent_pid = os.getpid ()         self._popen  = None        self._target = target         self._args = tuple (args)          self._kwargs = dict (Kwargs)         self._name  = name or type (self) .__name__ +  '-'  +                        ': ' Join (STR (i )  for i in self._identity)

Process ([group [, Target [, name [, args [, Kwargs]]])

Group is essentially not used, is a reservation and is easy to expand later.

Target represents the calling object,

Args represents the positional parameter tuple of the calling object

Kwargs represents the dictionary of the Calling object

Name is the alias, which is the name of the process

Its methods/properties are followed by threading. Thread also has a lot of similar places, mainly:

Start (): Start process activity.

Run (): Represents the active method of the process, which can be overridden in a subclass.

Join ([timeout]): is used to block the current context until the process is finished running, a process can be join () multiple times, the timeout unit is seconds.

Terminate (): End process. Using Sigterm on UNIX, use TerminateProcess on the Windows platform

Is_alive (): Determine if the process is still alive.

Name: A string that represents the name of the process, or it can be used by an assignment statement to modify the name of the process

Ident: ID of the process, if the process does not start, the result is None

PID: With ident, we can see the implementation of ident and PID, is the use of the OS Module Getpid () method.

Authkey: Sets/Gets the authorization password for the process. When initializing multiple processes, use Os.urandom () to assign a random string to the primary process. When a process object is created, it inherits the authentication key of its parent process, but can be changed by setting Authkey to another byte string. Here Authkey why can both set the authorization password and can get it? That's because its definition uses the property decorator, the source code is as follows:

@propertydef Authkey: Return self._authkey@authkey.setterdef authkey (self, Authkey): ' Set authorization ke Y of Process "' Self._authkey = authenticationstring (Authkey)

This is a high-level use of the property, if you understand it is also very simple, interested to see other information.

Daemon: A Boolean value that indicates whether the process is (True) No (False) is a daemon. It must be set before calling start (), or RuntimeError will be thrown. Its initial value inherits from the process that created it; the process is not a daemon, so all processes created in the process default daemon = False.

ExitCode: Returns the code when the process exits. The process runs with a value of none, and if it is –n, the signal N ends.

(1) A simple single-process example

#coding =utf-8import multiprocessingimport datetimeimport timedef worker (interval):     n = 5    while n > 0:         print  "the now is %s"% datetime.datetime.now ()          time.sleep (interval)          n -= 1if __name__ ==  "__main__":     p = multiprocessing . Process (target = worker, args =  (3,))     p.start () #开始进程       #p. Terminate () #结束进程      #p. Join (9) #阻塞当前上下文     print   "P.authkey", p.authkey# gets the authorization password for the process     p.authkey = u "123" #设置进程的授权密码      print  "P.authkey",  p.authkey# gets the authorization password for the process     print  "P.pid:",  p.pid,p.ident# Process id    p.name =  ' HelloWorld ' #修改进程名字     print  "p.name:",  p.name# process Name     print  "p.is_alive:",  p.is_alive () #是否是活的

Running results such as:

650) this.width=650; "Src=" Http://s3.51cto.com/wyfs02/M01/8A/1D/wKioL1goCjnAajjrAAA45_WP-C4616.png-wh_500x0-wm_3 -wmp_4-s_964768909.png "title=" Selection _001.png "alt=" Wkiol1gocjnaajjraaa45_wp-c4616.png-wh_50 "/>

The above code has two lines commented out, you can remove the comments, understand the usefulness of these two methods, this does not stick to my running results.

(2) Customizing the Process class and opening multiple processes

Import multiprocessingimport datetimeimport timeclass myprocess (multiprocessing. Process):     "" "     Custom Process Class     " "     def __init__ (self,interval,group=none,target=none,name=none,args= (), kwargs={}):         multiprocessing. process.__init__ (Self,group,target,name,args,kwargs=kwargs)          Self.interval = interval    def run (self):         n = 5        while n > 0 :             print ("the time is %s" %datetime.datetime.now ())             time.sleep ( Self.interval)             n -= 1def worker_1 (interval):    print  "worker_1"      Time.sleep (interval)     print  "end worker_1" def worker_2 (interval):     print  "worker_2"     time.sleep (interval)      print  "end worker_2" def worker_3 (interval):    print  "worker_3"      time.sleep (interval)     print  "End worker_3" if __name__  ==  "__main__":     p1 = myprocess (Interval=2,target = worker _1, args =  (2,))     p2 = myprocess (interval=2,target =  worker_2, args =  (3,))     p3 = myprocess (interval=2,target =  worker_3, args =  (4,))     p1.start ()     p2.start ( )    &NBSp;p3.start ()     print  "Current process", multiprocessing.current_process (), Multiprocessing.active_children ()     print ("The number of cpu is:"  + str (Multiprocessing.cpu_count ()))     for p in  Multiprocessing.active_children ():         print ("child    p.name: " + p.name + " \tp.id " + str (p.pid))     print   "END!!!!!!!!!!!!!!!!!"

The results of the operation are as follows:

650) this.width=650; "Src=" Http://s4.51cto.com/wyfs02/M00/8A/2A/wKioL1gpxPeBX_kSAAC2R3nTomo506.png-wh_500x0-wm_3 -wmp_4-s_2666770130.png "title=" Selection _002.png "alt=" Wkiol1gpxpebx_ksaac2r3ntomo506.png-wh_50 "/>

Take a look at the printed time, three processes should be executed in parallel.

(ii) Inter-process communication

The multiprocessing module supports communication between two processes: Queue and pipe (pipeline).

(1) Queue

The queue class in multiprocessing is defined in the queues.py file. Similar to Queue.queue, the queue class in multiprocessing implements most of Queue.queue's methods (refer to the previous blog post python: threading, Process and co-processes (3)--queue module and source analysis), but task_ Done () and join () are not implemented, the main methods and properties are

Qsize (): Returns the size of the queue

Empty (): Returns a Boolean value that indicates whether the queue is empty

Full (): Returns a Boolean value that indicates whether the queue is filled

Put (item[, block[, timeout]): When you add an element to the queue Item,block set to False, the full exception is thrown if the queue is filled. If the block is set to True,timeout set to none, it will be added to the queue when there is a vacancy, otherwise the full exception will be thrown according to the timeout value set by timeouts.

Put_nowait (item): Equivalent to put (item,false).

Get ([block[, timeout]]): Removes the element from the queue and returns the value of the element, if timeout is a positive number, it blocks the maximum timeout in seconds, and throws an empty exception if no items are available for that time.

Get_nowait (): Equivalent to get (False)

Close (): Indicates that the queue is not adding new elements

Join_thread (): Joins the background thread. This can only be used after you call Close (). It blocks until the background thread exits, ensuring that all data in the buffer has been flushed to the pipeline. By default, if the process is not the creator of the queue, it exits, and it tries to join the queue's background thread. The process can call Cancel_join_thread () to do

Cancel_join_thread (): Block Join_thread () in blocking, preventing the background thread from being automatically connected when the process exits, and Ken can cause data loss.

(2) Pipe

Pipe is not a class, it is a function, the function is defined in the connection.py in multiprocessing, the function prototype pipe (DUPLEX=TRUE),

Returns a pair of connection objects Conn1 and conn2 connected through a pipe.

If Duplex is true (the default value), the pipeline is bidirectional.

If Duplex is false, the pipeline is one-way: Conn1 can only be used to receive messages, CONN2 can only be used to send messages.

Pipe () The two connection objects returned represent both ends of the pipeline, each connecting object having the Send () and recv () methods (as well as other methods), sending and receiving messages, respectively. Here's a simple example, one that sends data, one that accepts data

#coding =utf-8import multiprocessingimport timedef proc1 (pipe):     "" "      Send Data      ""     while True:         for i in xrange (:       )      print  "send: %s"  % (i)              pipe.send (i) #发送数据              time.sleep (1) def proc2 (pipe):     "" "     receive data       "" "    while True:         print  "Proc2 rev:",  pipe.recv () #接受数据         time.sleep ( 1) if __name__ ==  "__main__":     pipe1,pipe2 = multiprocessing. Pipe () #返回两个连接对象     p1 = multiprocessing. Process (target=proc1, args= (Pipe1,))     p2 = multiprocessing. Process (target=proc2, args= (Pipe2,))     p1.start ()     p2.start ()     p1.join ()     p2.join ()

The results of the operation are as follows:

650) this.width=650; "Src=" Http://s3.51cto.com/wyfs02/M01/8A/2E/wKiom1gpyifSnYJyAABmdzZq03c015.png-wh_500x0-wm_3 -wmp_4-s_3233799382.png "title=" Selection _003.png "alt=" Wkiom1gpyifsnyjyaabmdzzq03c015.png-wh_50 "/>

(iii) Inter-process synchronization

Multiprocessing contains primitives that are equivalent to all synchronization primitives in threading, and it also has lock,rlock,even,condition,semaphore,boundedsemaphore. Usage is similar, they are defined in the synchronize.py file of the multiprocessing package, but in this case, it is interesting to refer to Python: threads, processes, and association (2)--threading modules related to the concept of understanding. If you understand the relevant concepts, use in the multiprocessing module is the same, see the following simple example, there are two processes to write to a file, in order to avoid access to the conflict, you can use the lock.

#coding =utf-8import multiprocessingdef worker_with (lock, f):    with  Lock: #Lock等对象也是支持上下文管理器协议的.         fs = open (f,  ' A + ')          n = 10        while n >  1:            fs.write ("Lockd acquired  via with\n ")             n -=  1        fs.close () Def worker_no_with (lock, f):     lock.acquire ()     try:        fs  = open (f,  ' A + ')         n = 10         while n > 1:       &nBsp;    fs.write ("lock acquired directly\n")              n -= 1        fs.close ()     finally:        lock.release () if __name__  ==  "__main__":     lock = multiprocessing. Lock () #定义锁     f =  "/home/liulonghua/files.txt"     w =  multiprocessing. Process (target = worker_with, args= (lock, f))     nw =  Multiprocessing. Process (target = worker_no_with, args= (lock, f))     w.start ()      nw.start ()     print  "End"

Multiprocessing provides an IPC (such as pipe and queue) that is not available in the threading package, and is more efficient. Pipe and queue should be given priority, and synchronization methods such as lock/event/semaphore/condition should be avoided (since they occupy no resources of the user process).

Multiple processes should avoid sharing resources. In multi-threading, we can share resources more easily, such as using global variables or passing parameters. In a multi-process scenario, the above method is not appropriate because each process has its own independent memory space. At this point we can share the resources by sharing the memory and the manager's method. However, this improves the complexity of the program and reduces the efficiency of the program because of the need for synchronization. The next blog post goes on to process sharing and process pooling.

Python: threads, processes, and co---multiprocessing (4) module (1)