Document directory
- Introduction
- Use thread queue
- Use multiple queues
Introduction
For Python, there is no lack of concurrency options. Its standard library includes support for threads, processes, and asynchronous I/O. In many cases, Python simplifies the use of various concurrency methods by creating high-level modules such as Asynchronous, thread, and sub-process. In addition to the standard library, there are also some third-party solutions, such as Twisted, Stackless, and process modules. This article focuses on the use of Python threads and uses some practical examples. Although many good online resources detail thread APIs, this article attempts to provide some practical examples to illustrate some common thread usage modes.
Global interpreter Lock indicates that the Python interpreter is NOT thread-safe. The current thread must hold a global lock for secure access to Python objects. Because only one thread can obtain the Python object/c api, the interpreter regularly releases and reacquires the lock every time it passes through 100 bytecode instructions. The interpreter can check the thread switching frequency through
sys.setcheckinterval()Function. In addition, the lock will be released and re-acquired based on potential blocking I/O operations. For more information, see
Gil and Threading StateAnd
Threading the Global Interpreter Lock. It should be noted that due to GIL, CPU-limited applications will not benefit from the use of threads. We recommend that you use processes or create processes and threads in a hybrid manner when using Python.
First, it is very important to clarify the differences between processes and threads. The difference between threads and processes is that they share status, memory, and resources. For a thread, this simple difference is both its advantage and Its disadvantage. On the one hand, threads are lightweight and easy to communicate with each other, but on the other hand, they also bring about various problems including deadlocks, contention conditions and high complexity. Fortunately, because of GIL and the queue module, it is much less complex to use the Python language than to use other languages.
Use Python threads
To continue learning the content in this article, I assume that you have installed Python 2.5 or later, because many examples in this article will use the new features of the Python language, these features only appear after Python2.5. To start using a Python thread, we will start with a simple "Hello World" Example:
#! /usr/bin/env python#coding=utf-8import threadingimport datetimeclass ThreadClass(threading.Thread): def run(self): now = datetime.datetime.now() print "%s says Hello World at time: %s" % (self.getName(),now)for i in range(2): t = ThreadClass() t.start()
Result:
Thread-1 says Hello World at time: 14:43:26. 981173Thread-2 says Hello World at time: 14:43:26. 981375 observe the output result carefully. You can see that the Hello World statement is output from both threads with a date stamp. If you analyze the actual code, you will find that there are two import statements. One statement imports the date and time module, and the other statement imports the thread module. ClassThreadClassInherited fromthreading.ThreadBecause of this, you need to define a run method to execute the code you want to run in this thread. The only thing to note in this run method is that,self.getName()Is a method used to determine the thread name.
The last three lines of code actually call this class and start the thread. If you pay attention to it, you will find that the actual startup thread ist.start(). Inheritance is taken into account when designing the thread module, and the thread module is actually built on the basis of the underlying thread module. In most casesthreading.ThreadInheritance is a best practice because it creates a regular API for Thread Programming.
Use thread queue
As mentioned above, when multiple threads need to share data or resources, the usage of threads may become complicated. The thread module provides many synchronization primitives, including semaphores, conditional variables, events, and locks. When these options exist, the best practice is to focus on using queues. In comparison, queues are easier to process, and thread programming is more secure, because they can effectively transmit all access to resources from a single thread, it also supports clearer and more readable design patterns, such"URL obtaining thread-based"
#! /usr/bin/env python#coding=utf-8import urllib2import timeimport Queueimport threadinghosts = ["http://yahoo.com", "http://baidu.com", "http://amazon.com","http://ibm.com", "http://apple.com"]queue = Queue.Queue()class ThreadUrl(threading.Thread): '''Theaded url grab''' def __init__(self,queue): threading.Thread.__init__(self) self.queue = queue def run(self): """docstring for run""" while True: # grabs host from Queue host = self.queue.get() #grabs urls of hosts and prints first 1024 bytes of page url = urllib2.urlopen(host) print url.read(1024) #signals to queue job is done self.queue.task_done()start = time.time()def main(): """docstring for main""" #spawn a poll of threads, and pass them queue instance for i in range(5): t = ThreadUrl(queue) t.setDaemon(True) t.start() #populate queue with data for host in hosts: queue.put(host) #wait on the queue until everything has been processed queue.join()main()print "Elapsed Time: %s" % (time.time() - start)
For this example, more code needs to be explained, but it is not much complicated than the first thread example because the queue module is used. This mode is common and recommended when using threads in Python. The procedure is described as follows:
1. Create a Queue. Queue () instance and fill it with data. 2. Pass the filled data instance to the Thread class, which is created by inheriting threading. Thread. 3. Generate a daemon thread pool. 4. Each time a project is retrieved from the queue and the data and run methods in the thread are used to execute the corresponding work. 5. After this task is completed, use the queue. task_done () function to send a signal to the completed queue of the task. 6. Performing the join operation on the queue actually means that the main program is exited when the queue is empty.
When using this mode, note that by setting the daemon thread to true, the main thread or program can exit only when the daemon thread is active. This method creates a simple way to control the program flow, because before exiting, you can perform the join operation on the queue or wait until the queue is empty. The queue module documentation details the actual processing process. See references:
join()
Keep blocked until all projects in the queue are processed. When a project is added to the queue, the total number of unfinished tasks increases. When the user thread calls task_done () to retrieve the project and complete all the work, the total number of unfinished tasks is reduced. When the total number of incomplete tasks is reduced to zero,join()The blocking status ends.
Use multiple queues
Because the mode described above is very effective, you can connect to the additional thread pool and queue for expansion, which is quite simple. In the preceding example, you only output the starting part of the Web page. In the next example, the complete Web page obtained by each thread is returned and the result is placed in another queue. Then, set another thread pool added to the second queue, and then perform corresponding processing on the Web page. The work in this example includes parsing Web pages using a third-party Python module named Beautiful Soup. To use this module, you only need two lines of code to extract the title tag of each accessed page and print it out, as shown in"Multi-queue Data Mining websites"Example:
#! /usr/bin/env python# coding: utf-8import Queueimport threadingimport urllib2import timefrom BeautifulSoup import BeautifulSouphosts = ["http://yahoo.com", "http://baidu.com", "http://amazon.com","http://ibm.com", "http://apple.com"]queue = Queue.Queue()out_queue = Queue.Queue()class ThreadUrl(threading.Thread): '''Threaded Url Grab''' def __init__(self,queue,out_queue): threading.Thread.__init__(self) self.queue = queue self.out_queue = out_queue def run(self): """grabs host from Queue""" host = self.queue.get() #grabs urls of hosts and then grabs chunk of webpage url = urllib2.urlopen(host) chunk = url.read() #place chunk into out_queuet self.out_queue.put(chunk) #signals to queue job is done self.queue.task_done()class DatamineThread(threading.Thread): '''Thread Url Grab''' def __init__(self, out_queue): threading.Thread.__init__(self) self.out_queue = out_queue def run(self): """grabs host from queue""" chunk = self.out_queue.get() #parse the chunk soup = BeautifulSoup(chunk) print soup.findAll(['title']) #signals to queue job is done self.out_queue.task_done()start = time.time()def main(): #spawn a pool of threads, and pass them queue instance for i in range(5): t = ThreadUrl(queue,out_queue) t.setDaemon(True) t.start() #populate queue with data for host in hosts: queue.put(host) for i in range(5): dt = DatamineThread(out_queue) dt.setDaemon(True) dt.start() # wait on the queue until everything has been processed queue.join() out_queue.join()main()print "Elapsed Time: %s" % (time.time()-start)
When analyzing this code, you can see that we have added another queue instance and passed the queue to the first thread pool class.ThreadURL. Next, for another thread pool classDatamineThread, Almost copies the same structure. In the run method of this class, obtain the Web page and text block from each thread in the queue, and then use Beautiful Soup to process the text block. In this example, use Beautiful Soup to extract the title tag of each page and print it out. This example can be easily applied to some more valuable application scenarios, because you have mastered the core content of basic search engines or data mining tools. One idea is to use Beautiful Soup to extract links from each page and navigate by them.
Summary
This article studies Python threads and describes best practices for using queues to reduce complexity, reduce minor errors, and improve code readability. Although this basic mode is relatively simple, it can be used to solve various problems by connecting the queue and thread pool. In the last section, you begin to study how to create a more complex processing pipeline that can be used as a model for future projects. The references section provides many excellent references on regular concurrency and threads.
Finally, it is important to note that the thread cannot solve all the problems. In many cases, it may be more appropriate to use the process. In particular, when you only need to create many sub-processes and listen for the response, the standard library sub-process module may be easier to use. For more official instructions, see references.
This article references: http://www.cnblogs.com/slider/archive/2012/06/20/2556256.html