have been lazy to write about Python-related articles, just one day need to give children's shoes a simple lesson, rushed out of the birth of this article.
I was going to have a full chat today about high-performance concurrency, but the weekend is coming. So I took one of the points to introduce, about other aspects, interested in small partners can communicate with me. Talk about high-efficiency concurrency, often out of the following three scenarios:
- Process: Each logical control flow is a process that is scheduled and maintained by the kernel. Because the process has a separate virtual address space, and other control flow letters must rely on the displayed interprocess communication, which is what we call the IPC mechanism
- Threads: Threads should be the most familiar to us. It is essentially a logical flow running in a single process context that is dispatched by the kernel.
- I/O multiplexing: Applications explicitly dispatch their own logical flows in the context of a process. The logical flow is modeled as a state machine, and after the data arrives at the file descriptor, the main program explicitly transitions from one state to another. Because the program is all in a separate process, So all streams share the same address space. The basic idea is to use the Select function to require the kernel to suspend a process, and only one or more I/O events occur before the control is returned to the application
It may seem incomprehensible, but fortunately Python's support for all three aspects of the response simplifies our operations. Well, let's talk about one of these today-threads. On the one hand, considering the notion that most people are wired, on the other hand, they are more lightweight than process threads. Threads are easier to understand than the association process. The relationship between processes and threads can be expressed in the simplest of clothes:
The state of the thread
Any language that supports threading can have the following operating states, whether you do Java,python or C, first look at the following picture:
Here I simply explain the meanings of these states:
- NEW: The first step in using a thread is to create a thread, and the created thread simply enters the executable state, which is runnable
- Runnable: The thread that entered this state has not yet started running, and once the CPU allocates time slices to the thread, the thread will officially start running
- Running: The thread officially starts running, and during the run it may go into a blocked state, i.e. blocked
- Blocked: In this state, the thread pauses to run, and after unblocking, the thread enters the runnable state, waiting for the CPU to allocate the time slice to it again
- End: The thread method has finished executing or is returned because of an abnormal termination
It's all about life, birth-learning (preparation before work)-work-vacations
The most complex of these is that the thread enters the blocked state from running, typically in three cases:
- Sleep: After a thread actively calls the sleep () or join () method.
- Wait: The Wait () method is called in the thread, and additional threads are required to wake up through the Notify () method.
- Synchronization: Gets the lock in the thread, but because the resource is already occupied by another thread.
Until now, we have a basic concept of threading, light says no practice fake bashi, below we are three small examples to talk about the use of threads and the final two concepts in the thread: synchronization and communication.
Thread simple to use
There are two ways to implement multithreading in Python: One is to use a low-level _thread module and another advanced threading module, which I recommend using the threading module. Before we get started, let's start by understanding what common classes the Threading module provides to me:
Thread,lock,rlock,condition,event,semaphore,timer and Local.
These classes are a powerful tool in the development of multi-threading. But in space, we don't start talking.
We look directly at how to use multi-threading, which is crucial, as the old saying goes: To let the child run first let him learn to walk. We're going to go two steps:
import threading#具体做啥事,写在函数中def run(number): ‘\n‘) print(number)if‘__main__‘: forin range(10): #注意这,开始咯,指明具体的方法和方法需要的参数 my_thread = threading.Thread(target=run, args=(i,)) #一定不要忘记 my_thread.start()
Multi-threaded creation and operation are the routines ah, write more natural ripe, to see the results of the operation:
thread-1,value=0thread-2,value=1thread-3,value=2thread-4,value=3thread-5,value=4thread-6,value=5thread-7,value=6thread-8,value=7thread-9,value=8thread-Ten,value=9
Synchronization and communication
The hardest problem in multithreaded development is not how to use it, but how to write the right and efficient code, and to write the right and efficient code must understand two very important concepts: synchronization and communication.
The so-called communication refers to how the messages are exchanged between threads, and synchronization is used to control the relative order in which operations occur between different threads. Simply put, synchronization is the order in which multiple threads access code, and communication is how messages are delivered between threads. The simplest solution to synchronizing in Python is to use the lock mechanism. The simplest solution for communication is the event. Here's a look at the specific use of both.
Thread synchronization
Competition occurs when multiple threads access the same resource at the same time, which is a bit like a lot of men chasing the same sister paper, and the results are unpredictable. So it is necessary to use a mechanism to ensure that every boy has the opportunity to get along with the girls, which is kind of like putting the little girl in a house, and then entering the boys locked the door The next guy wants to go in, he has to wait for a guy to come out. Just call it a lock.
The Python threading module provides us with the thread lock function, provides the Rlock object in the threading, and maintains a lock object inside the Rlock object, which is a kind of reentrant lock. For a lock object, if a thread acquire the operation two times in a row, the second acquire will suspend the thread because there is no release after the first acquire. This causes the lock object to never release, causing the thread to deadlock. The Rlock object allows a thread to acquire its operations multiple times, because the number of threads acquire is maintained internally through a counter variable. And each time the acquire operation must have a release operation corresponding to it, after all the release operation is completed, the other thread can request the Rlock object.
Through the lock mechanism, the process of accessing the shared resource in the final multithreading is similar to the following:
In fact, it demonstrates the essence of using locks to resolve thread synchronization: Serialize all threads to read and write operations on shared resources.
Also give a simple example to illustrate the simplest usage of rlock:
ImportThreadingmylock = Threading. Rlock () num =0 class workthread(threading. Thread): def __init__(self, name):Threading. Thread.__init__ (self) self.t_name = name def run(self): GlobalNum while True: Mylock.acquire () print (' \n%s locked, Number:%d '% (Self.t_name, num))ifNum >=4: Mylock.release () print (' \n%s released, Number:%d '% (Self.t_name, num)) Breaknum + =1Print' \n%s released, Number:%d '% (Self.t_name, num)) mylock.release () def test():Thread1 = Workthread (' A-worker ') Thread2 = Workthread (' B-worker ') Thread1.start () Thread2.start ()if__name__ = =' __main__ ': Test ()
To see the results of the operation:
A-worker locked, Number:0A-worker released, Number:1A-worker locked, Number:1A-worker released, Number:2A-worker locked, Number:2A-worker released, Number:3A-worker locked, Number:3A-worker released, Number:4A-worker locked, Number:4A-worker released, Number:4B-worker locked, Number:4B-worker released, Number:4
Some alumni asked if there are other ways to achieve similar effects except lock and rlock, of course, such as condition and semaphore have similar functions, of which condition is re-packaged on the basis of Lock/rlock, The semaphore principle is consistent with the operating system's PV operation. The reason for not elaborating is that there is no essential difference between the basic use and the principle. I personally have always thought that the more complex things behind the more simple principles, of course, welcome interested students and me to explore.
Thread Communication
In many cases, we need to pass messages between threads, also known as thread communication. The event provided in Python is one of the simplest communication mechanisms. Using threading.event allows a thread to wait for notifications from other threads, and we pass this event to the thread object, which has a flag with an initial value of false. Once the thread enters the waiting state through the wait () method until another thread calls the event's set () method to set the built-in flag to true, the event notifies all waiting-state threads to resume running. Let's take a look at some of the commonly used methods in event:
| Method Name |
meaning |
| IsSet () |
Test if the built-in identity is true |
| Set () |
Set the identity to True and notify all blocked threads to resume running |
| Clear () |
Set the identity to False |
| Wait ([timeout]) |
If the identity is true, returns immediately, otherwise blocks the thread to a blocking state, waiting for other threads to call set () |
To take a look at a simple example, let's say that you have 6 sister papers to get them up, so what are you going to do?
ImportThreadingImportTime class workthread(threading. Thread): def __init__(self, signal):Threading. Thread.__init__ (self) Self.singal = signal def run(self):Print"sister paper%s, slept ..."% self.name) self.singal.wait () print ("sister paper%s, get up ..."% self.name)if__name__ = ="__main__": Singal = Threading. Event () forTinchRange0,6): Thread = Workthread (Singal) Thread.Start () print ("Wake up in three seconds.") Time.sleep (3)#唤醒阻塞中的妹纸Singal.set ()
Here you act as the main thread, each sister paper is a sub-thread, no accident three seconds later you will wake up on time all the sister paper:
Sister paper thread-1, slept....Sister paper thread-2, slept....Sister paper thread-3, slept....Sister paper thread-4, slept....Sister paper thread-5, slept....Sister paper thread-6, slept....Three seconds, call sister paper, get up sister paper thread-1Get up...Sister paper thread-2Get up...Sister paper thread-5Get up...Sister paper thread-4Get up...Sister paper thread-3Get up...Sister paper thread-6Get up...
It is possible to use the event to implement thread communication, but another way to thread communication is through queues, which are queue. A thread-safe queue is provided in the standard library of Python, based on FIFO (first-in-one-out) implementations, which facilitates the delivery of messages between threads. The use is very simple, its principle is not difficult, with a simple diagram to show:
In addition, the multi-threaded communication process which conforms to this kind of structure is called producer-consumer model.
Thread pool
In fact, the use of multithreading is very simple, more is based on the specific business situation to write the corresponding logic. For the first time, considering the processor's resources is limited, can not blindly create threads, I have seen some small partners when writing crawlers, 100 URLs created 100 threads, The consequences are conceivable, so when there is a need to use a lot of threads, consider using thread pooling technology.
In addition, I only recommend the use of multithreading for processing about I/O operations, or it would cause performance degradation
Python multithreading Getting Started Guide