python-Multithreaded Learning

Source: Internet
Author: User

Disclaimer: This article is organized from "Curved Pomelo blog", "Python: Using threading Module to achieve multithreaded Programming" series, the original author for Walfred, Whypro, out of respect for the author, reprint, please indicate the original author.

First, the summary

Python's explanatory language also has a dedicated threading model, and Python virtual machines use the Gil (Global interpreter lock, globe interpreter Lock) to mutually exclusive threads access to shared resources, but are temporarily unable to take advantage of multiprocessor benefits.

In Python we are mainly through the thread and threading these two modules, wherein the Python threading module is to do some packaging thread, can be more convenient to use, so we use The threading module implements multithreaded programming. This article mainly looks at Python's support for multithreaded programming.

At the language level, Python provides very good support for multithreading, allowing you to easily support the creation of threads, mutexes, semaphores, and synchronization. The following is the official online Introduction threading module basic information and functions:

Implementation module

Thread: Multithreading of the underlying support module, generally not recommended to use;

Threading: Thread is encapsulated, and some threads are manipulated.

Threading Module

Thread threading class, this is the most we use a class, you can specify the thread function execution or inherit from it can implement sub-threading function;

A timer is similar to a thread, but waits for a period of time before it starts to run;

Lock the primitive, which we can use when the global variable is mutually exclusive;

Rlock can re-enter the lock, so that the single-threaded can obtain the acquired lock again;

Condition A conditional variable that allows a thread to stop waiting for another thread to satisfy a certain "condition";

The condition variable that is common to the Event. Multiple threads can wait for an event to occur, and all threads are activated after the event occurs;

Semaphore provides a structure similar to the "Waiting Room" for the thread that waits for the lock;

Boundedsemaphore is similar to semaphore, but is not allowed to exceed the initial value;

Queue: Implements multi-producer (Producer), multi-Consumer (Consumer) queues, supports lock primitives, and provides good synchronization support across multiple threads.

where the thread class

Is your primary threading class and you can create process instances. The functions provided by this class include:

GetName (self) Returns the name of the thread

The IsAlive (self) Boolean flag that indicates whether the thread is still running

Isdaemon (self) Returns the daemon flag for the thread

The Join (self, timeout=none) program hangs until the thread ends and, if a timeout is given, blocks timeout seconds

Run (self) defines the function function of the thread

Setdaemon (self, daemonic) sets the thread's daemon flag to Daemonic

SetName (self, name) sets the name of the thread

Start (self) starts thread execution

Where queue provides the class

Queue queues

Lifoqueue after in first out (LIFO) queue

Priorityqueue Priority Queue

Two ways of threading

In Python we are mainly through the thread and threading these two modules to achieve, wherein the Python threading module is to do some packaging thread, can be more convenient to be used, So we use the threading module for multithreaded programming. In general, there are two modes of using threads, one is to create a function to execute the thread, pass the function into the thread object, let it execute, and the other is to inherit from thread directly, create a new class, and put the code executed by the thread into the new class.

Passing a function into a thread object

"'  Created on 2012-9-5     @author:  walfred @module: Thread. ThreadTest1  @description: '    import threading    def thread_fun (num): for-      N in range (0, int (num)):          Print "I come from%s, num:%s"% (Threading.currentthread (). GetName (), N)    def main (thread_num):      thread_list = li St ();      # first create thread object      for I in range (0, thread_num):          thread_name = "thread_%s"%i          thread_list.append (threading. Thread (target = thread_fun, name = Thread_name, args = ())            # starts all threads for thread in         thread_list:          Thread.s Tart ()            # waits for all child threads in the main thread to exit for      the thread in Thread_list:          thread.join ()    if __name__ = = "__main__":      Main (3)  

The program started 3 threads, and printed the thread name of each thread, this is relatively simple, to deal with repetitive tasks to come in handy, the following describes the use of inherited threading way;

Inherit from threading. Thread class

"' Created on 2012-9-6   @author: walfred @module: Thread. ThreadTest2 '    import threading    class MyThread (threading. Thread):      def __init__ (self):          threading. Thread.__init__ (self);                def run:          print "I am%s"%self.name            if __name__ = = "__main__": for      Thread in range (0, 5):          t = Mythre AD ()          T.start ()  

Third, threading. Important functions of the thread class

This article mainly introduces some main methods of the main class thread in the threading module, the example code is as follows:

"'  Created on 2012-9-7     @author:  walfred @module: Thread. ThreadTest3  @description: '    import Threading    class MyThread (threading. Thread):      def __init__ (self):          threading. Thread.__init__ (self)            def run (self):          print "I am%s"% (self.name)        if __name__ = = "__main__": For      I in R Ange (0, 5):          My_thread = MyThread ()          My_thread.start ()    

Name related

You can specify a name for each thread, the default is Thread-no, as shown in the example code above:

I am thread-1i am thread-2i am thread-3i am thread-4i am Thread-5

Of course you can specify the name of each thread, this through the SetName method, the code:

def __init__ (self):      threading. Thread.__init__ (self)      

Join method

The Join method is prototyped as follows, and this method is used to block the current context until the thread finishes running:

def join (self, timeout=none):  

Timeout to set time-out parameters

Setdaemon method

When we execute a main thread in a program, if the main thread creates a sub-thread, the main thread and the child thread are suited two ways, and when the main thread finishes trying to exit, it verifies that the child thread is complete. If the child thread is not completed, the main thread waits for the child thread to complete before exiting. But sometimes what we need is, as long as the main thread is complete, regardless of whether the child thread is complete, and the main thread to exit, then you can use the Setdaemon method, and set its parameter to True.

Iv. using lock Mutex

Now let's consider the question: what should our code say if each thread needs access to the same public resource?

"' Created on 2012-9-8   @author: walfred @module: Thread. ThreadTest3 '  import threading Import Time     counter = 0     class MyThread (threading. Thread):      def __init__ (self):          threading. Thread.__init__ (self)            def run (self):          Global Counter          time.sleep (1);          Counter + = 1          print "I am%s, set counter:%s"% (self.name, counter)        if __name__ = = "__main__": For      I in ran GE (0, $):          My_thread = MyThread ()          My_thread.start ()     

To solve the above problem, we might write this code, we assume that running 200 threads, but the 200 threads will go to access counter this public resource, and the resource processing (counter + = 1), the code looks like this, but we look at the results of the operation:

I am Thread-69, set counter:64i am Thread-73, set counter:66i am Thread-74, set counter:67i am Thread-75, set counter:68i  Am Thread-76, set counter:69i am Thread-78, set counter:70i am Thread-77, set counter:71i am Thread-58, set counter:72i am THREAD-60, set counter:73i am Thread-62, set counter:74i am Thread-66, set counter:75i am Thread-70, set counter:76i am T hread-72, set counter:77i am Thread-79, set counter:78i am Thread-71, set counter:78

Printing results I only posted a part, from which we have seen that the global resource (counter) is preempted, the problem is caused by not controlling the access of multiple threads to the same resource, causing damage to the data, so that the results of the thread run is not predictable. This behavior is called "Thread insecurity." In the development process we have to avoid this situation, how to avoid it? This uses the mutex we mentioned in the review.

The concept of Mutex lock

In Python programming, the concept of object mutexes is introduced to ensure the integrity of shared data operations. Each object corresponds to a tag that can be called a "mutex", which is used to guarantee that only one thread can access the object at any one time. In Python we use the lock class provided by the threading module.

We have to rectify the above program, for which we need to add a mutex variable mutex = threading. Lock (), and then we will preempt this lock mutex.acquire () before we scramble for resources, and we release this lock Mutex.release () after the use of the resource is complete. The code is as follows:

"' Created on 2012-9-8   @author: walfred @module: Thread. ThreadTest4 '    import threading Import Time     counter = 0  Mutex = Threading. Lock ()     class MyThread (threading. Thread):      def __init__ (self):          threading. Thread.__init__ (self)            def run (self):          Global Counter, Mutex          time.sleep (1);          If Mutex.acquire ():              counter + = 1              print "I am%s, set counter:%s"% (self.name, counter)              mutex.release () 
   if __name__ = = "__main__":      for I in range (0, +):          My_thread = MyThread ()          My_thread.start ()        

Synchronous blocking

When a thread calls the lock object's acquire () method to obtain the locks, the lock enters the "locked" state. Because only one thread 1 can acquire a lock at a time, if another thread 2 attempts to acquire the lock at this point, the thread 2 becomes "blo synchronous blocking state." The lock enters the "unlocked" state until the lock is released by the release () method that has the lock's thread 1 call lock. The thread scheduler chooses one of the threads in the synchronous blocking state to obtain the lock and causes the thread to enter the running (running) state.

Further consideration

By using mutual exclusion locks on public resources, it is easy to reach our goal, but if we encounter the following situation:

1, how to encounter lock nesting situation, this nesting refers to when I a thread in the acquisition of critical resources, but also need to get again;

2, if there are multiple public resources, the thread sharing multiple resources, if the two threads occupy a portion of the resources and waiting for each other's resources;

These two situations will directly cause the program to hang, that is, deadlock, below we talk about deadlock and can be re-entry lock Rlock.

V. Formation of the deadlock

If you have more than one public resource, when you share multiple resources between threads, what would be the problem if the two thread had a portion of the resources and waited for each other's resources?

Deadlock concept

The so-called deadlock: refers to two or more than two processes in the execution process, because of the competition for resources caused by a mutual waiting phenomenon, if there is no external force, they will not be able to proceed. At this point the system is in a deadlock state or the system generates a deadlock, and these processes, which are always waiting on each other, are called deadlock processes. Because the resource consumption is mutually exclusive, when a process requests resources, so that the process without external assistance, never allocated the necessary resources and can not continue to run, which creates a special phenomenon of deadlock.

"' Created on 2012-9-8 @author: walfred @module: Thread. TreadTest5 ' Import Threading Countera = 0 Counterb = 0 mutexa = Threading. Lock () Mutexb = Threading. Lock () class MyThread (threading. Thread): def __init__ (self): threading.          Thread.__init__ (self) def run (self): self.fun1 () self.fun2 () def fun1 (self): Global Mutexa, Mutexb if Mutexa.acquire (): print "I am%s, get res:%s"% (Self.name, "ResA"                  If Mutexb.acquire (): print "I am%s, get res:%s"% (Self.name, "Resb") Mutexb.release () mutexa.release () def fun2 (self): global mutex                            A, Mutexb if Mutexb.acquire (): print "I am%s, get res:%s"% (Self.name, "Resb") If Mutexa.acquire (): print "I am%s, get res:%s"% (Self.name, "ResA") mutexa. RelEase () mutexb.release () if __name__ = = "__main__": For I in range (0): My_       thread = MyThread () My_thread.start ()

The code shows the two functional functions of a thread that acquire another competing resource after acquiring a competing resource, and we look at the result of the operation:

I am Thread-1, get Res:resai am Thread-1, get res:resbi am Thread-2, get Res:resai am Thread-1, get res:resb

As you can see, the program has been suspended there, and this is what we call a "deadlock."

Avoid deadlocks

The main method to avoid deadlock is: the correct and orderly allocation of resources, to avoid the deadlock algorithm is the most representative algorithm Dijkstra E.W in 1968 proposed banker algorithm.

Six, can be re-entry lock Rlock

Consider this: What to do if a thread encounters a lock nesting, which means that when one of my threads acquires a critical resource, it needs to be fetched again.

According to this situation, the code is as follows:

"' Created on 2012-9-8   @author: walfred @module: Thread. ThreadTest6 '    import threading Import Time     counter = 0  Mutex = Threading. Lock ()     class MyThread (threading. Thread):      def __init__ (self):          threading. Thread.__init__ (self)            def run (self):          Global Counter, Mutex          time.sleep (1);          If Mutex.acquire ():              counter + = 1              print "I am%s, set counter:%s"% (self.name, counter)              if Mutex.acquire ():                  counter + = 1                  print "I am%s, set counter:%s"% (self.name, counter) mutex.release ()              mutex.release ()        if __name__ = = "__main__":      for I in range (0, $):          My_thread = MyThread ()          My_thread.start ()         

The code for this situation is running as follows:

I am Thread-1, set counter:1

After that, it hangs directly, and this situation forms the simplest deadlock.

Is there a situation where a particular thread can access a competing resource with a mutex? In Python, in order to support multiple requests for the same resource in the same thread, Python provides a "reentrant lock": Threading. Rlock. The Rlock internally maintains a lock and a counter variable, counter records the number of acquire, so that resources can be require multiple times. Until all the acquire of a thread are release, the other threads can get the resources. In the example above, if you use Rlock instead of lock, a deadlock will not occur:

The code simply takes the above:

Mutex = Threading. Lock ()  

Replace with:

Mutex = Threading. Rlock ()  

Can.

Vii. using condition for complex synchronization

We have now used lock to mutually exclusive access to public resources, but also discussed that the same thread can use Rlock to re-enter the lock, but even though we only deal with the simple synchronization of some programs, we are not even reasonable to solve the deadlock caused by using lock lock problem. So we have to learn to use a deeper approach to solving synchronization problems.

The condition object provided by Python provides support for complex thread synchronization issues. condition is called a conditional variable and provides the wait and notify methods in addition to the acquire and release methods similar to lock.

The main way to use condition is that the thread first acquire a condition variable and then determines some conditions. Wait if the condition is not met, and if the condition is met, after some processing changes the condition, the other thread is notified by the Notify method, and the other thread in the wait state will be re-judged after receiving the notification. Constantly repeating this process to solve complex synchronization problems.

Below we demonstrate with a well-known "producer-consumer" model, using condition in Python for complex synchronization.

"' Created on 2012-9-8 @author: walfred @module: Thread. TreadTest7 ' Import threading import time Condition = Threading. Condition () products = 0 Class Producer (threading. Thread): def __init__ (self): threading.              Thread.__init__ (self) def run (self): global condition, products while True:                      If Condition.acquire (): If products < 10:products + = 1;                  Print "Producer (%s):d Eliver One, now products:%s"% (self.name, Products) condition.notify ()                      Else:print "Producer (%s): Already, stop deliver, now products:%s"% (self.name, products)                  Condition.wait (); Condition.release () Time.sleep (2) class Consumer (threading. Thread): def __init__ (self): threading. Thread.__init__ (self) def run (self): Global condition, Products while True:if Condition.acquire (): If Products > 1:                      Products-= 1 print "Consumer (%s): Consume one, now products:%s"% (self.name, products) Condition.notify () Else:print "Consumer (%s): Only 1, stop consume, produ                  cts:%s "% (self.name, Products) condition.wait (); Condition.release () Time.sleep (2) if __name__ = = "__main__": for P in range (0, 2) : P = Producer () P.start () for C in range (0, ten): c = Consumer () C.star   T ()

The main implementation of the code in the producer and consumer threads, the two sides will be around the products to create synchronization problems, first of all, 2 generators to produce product, and then the next 10 consumers will be consumed, the code runs as follows:

Producer (Thread-1):d Eliver One, now Products:1producer (Thread-2):d Eliver One, now Products:2consumer (Thread-3): Consume one, now Products:1consumer (Thread-4): Only 1, stop consume, Products:1consumer (Thread-5): Only 1, stop consume, PR Oducts:1consumer (Thread-6): Only 1, stop consume, Products:1consumer (Thread-7): Only 1, stop consume, Products:1consumer (Thread-8): Only 1, stop consume, Products:1consumer (Thread-10): Only 1, stop consume, Products:1consumer (Thread-9): only 1, stop consume, Products:1consumer (Thread-12): Only 1, stop consume, Products:1consumer (Thread-11): Only 1, stop consume, Products:1

In addition: the constructor of the condition object can accept a Lock/rlock object as a parameter, and if not specified, the condition object creates a rlock on its own, except for the Notify method. The condition object also provides a Notifyall method that notifies all threads in the waiting pool to attempt to acquire an internal lock. As a result of the above mechanism, threads in the waiting state can only be awakened by the Notify method, so the role of notifyall is to prevent the thread from ever being in a state of silence.

Viii. using event to implement inter-thread communication

Use Threading. Event can be used to communicate between threads, we have initially implemented the basic functionality of inter-thread communication, but a more general approach is to use threading. The event object.
Use Threading. The event allows a thread to wait for notifications from other threads, and we pass this event to the thread object, which has a flag built in by default with an initial value of false. Once the thread enters the waiting state through the wait () method until another thread calls the event's set () method to set the built-in flag to true, the event notifies all waiting-state threads to resume running.

"' Created on 2012-9-9   @author: walfred @module: Thread. TreadTest8 '    import threading Import time    class MyThread (threading. Thread):      def __init__ (self, Signal):          Threading. Thread.__init__ (self)          Self.singal = Signal                def run (self):          print "I am%s,i'll sleep ..."%self.name          Self.singal.wait ()          print "I am%s, I awake ..."%self.name            if __name__ = = "__main__":      Singal = Threading. Event ()      for T in range (0, 3):          thread = MyThread (singal)          Thread.Start ()            print "Main Thread sleep 3 sec Onds ... "      Time.sleep (3)            Singal.set ()  

The results are as follows:

I am thread-1,i'll sleep ... I am thread-2,i'll sleep ... I am thread-3,i'll sleep ... main Thread sleep 3 seconds ... I am Thread-1, I awake ... I am Thread-2, I awake ... I am Thread-3, I awake ...

python-Multithreaded Learning

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.