Queue module and source code analysis

Source: Internet
Author: User
The queue module is a module that provides queue operations, which are the most commonly used interchange data between threads. The module provides three types of queues:

Queue.queue (MaxSize): FIFO, maxsize is the size of the queue, and its value is non-positive when it is a wireless loop queue

Queue.lifoqueue (maxsize): LIFO, equivalent to Stack

Queue.priorityqueue (MAXSIZE): Priority queue.

Where Lifoqueue,priorityqueue is the subclass of the queue. Three of them have the following common methods:

Qsize (): Returns the approximate queue size. Why do you want to add the word "approximate"? Because the Get () method is not blocked when concurrent execution is not guaranteed when the value is greater than 0, it is also valid for the put () method.

Empty (): Returns a Boolean value, returns True if the queue is empty, and returns false instead.

Full (): Returns True if the queue size is set, otherwise false.

Put (Item[,block[,timeout]]): When adding elements to the queue Item,block set to False, the full exception is thrown if the queue is filled. If the block is set to True,timeout set to none, it will be added to the queue when there is a vacancy, otherwise the full exception will be thrown according to the timeout value set by timeouts.

Put_nowwait (item): Equivalent to put (item,false). When the block is set to False, if the queue is empty, an empty exception is thrown. If the block is set to True,timeout set to none, it will be added to the queue when there is a vacancy, otherwise the empty exception will be thrown according to the timeout value set by timeouts.

Get ([Block[,timeout]): Removes the element from the queue and returns the value of the element, if timeout is a positive number, it blocks the maximum time-out in seconds, and throws an empty exception if no items are available for that time.

Get_nowwait (): Equivalent to get (False)

Task_done (): Sends a signal that the into row task is complete and is often used in consumer threads.

Join (): blocks until all elements of the queue are processed, and then other operations are processed.

(i) Source code analysis

The queue module is very simple and simple to use, but I think it is necessary to put the relevant source code of the module analysis, will learn a lot of things, look at the code of the Great God write how beautiful, how the structure of the modular, and then think of their own code, are tears, to learn to learn. In order to shorten the length, the source code of the comment part is deleted.

From time import time as _timetry:import threading as _threadingexcept importerror:import dummy_threading as _thre Adingfrom Collections Import Dequeimport HEAPQ __all__ = [' Empty ', ' full ', ' Queue ', ' priorityqueue ', ' lifoqueue '] class Em    Pty (Exception): "Exception raised by Queue.get (block=0)/get_nowait ()."    Pass class Full (Exception): "Exception raised by Queue.put (block=0)/put_nowait ()." Pass class Queue:def __init__ (Self, maxsize=0): Self.maxsize = maxsize self._init (maxsize) self.m Utex = _threading. Lock () Self.not_empty = _threading. Condition (Self.mutex) self.not_full = _threading. Condition (Self.mutex) Self.all_tasks_done = _threading.  Condition (Self.mutex) self.unfinished_tasks = def get_nowait (self): return Self.get (False) def _init (self, maxsize): Self.queue = Deque () def _qsize (self, Len=len): Return Len (self.queue) def _put  (Self, item): Self.queue.append (item)  def _get (self): return Self.queue.popleft () 

Through the analysis of several functions later, the queue object is based on the queue of the collections module (about Collections module Reference Python: Using counter for counting statistics and collections modules), Plus the threading module mutex and the conditional variable are encapsulated.

Deque is a double-ended queue that works well for queues and stacks. The queue object above is a first-in, out-of-the-line, so first the _init () function defines a double-ended queue, and then it defines the _put () and _get () functions, respectively, by adding elements to the right of the double-ended queue, deleting the elements on the left, which constitutes a FIFO queue, Similarly it is easy to think of the implementation of Lifoqueue (LIFO queue), to ensure that the right side of the queue to add the right to delete. You can post the source code to see.

Class Lifoqueue (queue):    ' Variant of the Queue that retrieves most recently added entries first. '     def _init (self, maxsize):        self.queue = []     def _qsize (self, Len=len):        return len (self.queue)     def _put ( Self, item):        self.queue.append (item)     def _get (self):        return Self.queue.pop ()

Although its "queue" is not used in queue (), the same list is used because the list append () and the pop () action are to add elements to the rightmost and delete the rightmost element.

Looking at Priorityqueue, he is a priority queue, which uses the Heappush () and Heappop () two functions of the HEAPQ module. The HEAPQ module provides a modular approach to the data structure of the heap, which can be built, and the HEAPQ modules provide the appropriate methods to operate the heap. where the _init () function self.queue=[] can be seen as creating an empty heap. Heappush () Inserts a new value into the heap, Heappop () pops the minimum value from the heap, which allows for precedence (this is a simple introduction to the HEAPQ module). The source code is as follows:

Class Priorityqueue (queue):    ' Variant of the queue that retrieves open entries in the priority order (lowest first).     Entries is typically tuples of the form:  (priority number, data).    "     def _init (self, maxsize):        self.queue = []     def _qsize (self, Len=len):        return len (self.queue)     def _put ( Self, item, Heappush=heapq.heappush):        heappush (Self.queue, item)     def _get (self, Heappop=heapq.heappop):        return Heappop (self.queue)

The basic data structure is analyzed, and then the other parts are analyzed.

A mutex is a threading. The lock () object is a mutex, and the three Not_empty, Not_full, and All_tasks_done are threading. Condition () object, condition variable, and maintain the same lock object mutex (for the lock object and Condition object in the threading module refer to the previous blog post, Python: Thread, Process, and coprocessor (2)--threading module).

which

Self.mutex Mutex: Any operation that gets the state of the queue (empty (), qsize (), and so on), or modifies the contents of the queue (Get,put, etc.) must hold the mutex. Acquire () Gets the lock, release () releases the lock. At the same time, the mutex is maintained by three condition variables.

Self.not_empty condition variable: When a thread adds data to a queue, it calls Self.not_empty.notify () to notify other threads, and then wakes up a thread that removes the element.

Self.not_full condition Variable: When an element is removed from the queue, a thread that adds the element is awakened.

Self.all_tasks_done condition Variable: When the number of unfinished tasks is deleted to 0 o'clock, notifies all tasks to complete

Self.unfinished_tasks: Define the number of unfinished tasks


Take a look at the main methods:

(1) put ()

The source code is as follows:

def put (self, item, block=true, Timeout=none): Self.not_full.acquire () #not_full获得锁 try:                    If self.maxsize > 0: #如果队列长度有限制 if not block: #如果没阻塞 If self._qsize () = = Self.maxsize: #如果队列满了抛异常 Raise full elif timeout I S None: #有阻塞且超时为空, wait while self._qsize () = = Self.maxsize:self.not_ful L.wait () elif Timeout < 0:raise valueerror ("' Timeout ' must be a non-negative number"                    else: #如果有阻塞, and timeout non-negative, end time = current time + Timeout Endtime = _time (+) + timeout While self._qsize () = = Self.maxsize:remaining = Endtime-_time () if REM aining <= 0.0: #到时后, throw an exception raise full #如果没到时, the queue will always be suspended until there is "                   Location "Frees up     Self.not_full.wait (remaining) self._put (item) #调用_put方法, adding elements self.unfinished _tasks + = 1 #未完成任务 +1 self.not_empty.notify () #通知非空, wake up non-empty suspended tasks Finally:sel F.not_full.release () #not_full释放锁

By default, the block is true,timeout to none. If the queue is full, then the _put method is called to add the process to the deque (described later), and an incomplete task plus 1 also notifies the queue to be non-empty.

If the block parameter is set to Flase, an exception will be thrown when the queue is full. If a timeout is set, then the time-to-date blocking, time-to-throw exception. This method operates using the Not_full object.

(2) Get ()

The source code is as follows:

def get (self, block=true, Timeout=none): Self.not_empty.acquire () #not_empty获得锁 try:                    If not block: #不阻塞时 if not self._qsize (): #队列为空时抛异常                    Raise empty Elif timeout is None: #不限时时, the queue is empty and waits for the while Not Self._qsize (): Self.not_empty.wait () elif Timeout < 0:raise valueerror ("' Timeout ' must b E a non-negative number ") Else:endtime = _time () + timeout while not self._qsize (): remaining = Endtime-_time () if remaining <= 0.0:rai Se Empty self.not_empty.wait (remaining) item = Self._get () #调用_get方法, removing and obtaining entries             Mesh self.not_full.notify () #通知非满 return item #返回项目 finally: Self.not_empTy.release () #释放锁 

Logic is the same as the put () function, which, by default, waits for the queue to empty, otherwise it will call the _get method (see below) to remove and get an item, and finally return the item. This method operates using the Not_empty object.

But I think the put () and get () two functions together to understand better. Not_full and Not_empty represent two different types of operation of the thread, Not_full can be understood as is-not-full, that is, the queue is full, the default is not full, no full time not_full This condition variable to obtain the lock, and do some conditions to judge, Only the conditions can be added to the queue to add elements, the addition will notify the NOT_EMPTY condition variable queue is not empty, "I" has just added an element to meet the basic conditions that can perform the delete action (the queue is not empty, think if it is empty to perform delete action is meaningless), At the same time, wake up some of the suspended thread to perform the removal action, let these threads re-determine the condition, if the condition is allowed to perform the delete action, and then notify the not_full condition variable, tell "it" queue is not full, because "I" just deleted an element (think if the queue is full of added elements will not be added , it makes no sense), satisfies the basic condition of adding elements (the queue is not full), and wakes up some of the threads that are suspended to perform the add action, these threads will then be conditionally judged, the elements will be added if the conditions are met, otherwise the hang, and so on, and so on, also ensure the security of the thread. As I said earlier, when an element is removed from the queue, it wakes up a thread that adds the element, and when an element is added, it wakes up a thread that deletes the element.

(3) Task_done ()

The source code is as follows:

def task_done (self):            self.all_tasks_done.acquire ()       #获得锁        try:            unfinished = self.unfinished_tasks-1  #判断队列中一个线程的任务是否全部完成            If unfinished <= 0:                     #是则进行通知, or in an overdose call the Times exception                if unfinished < 0:                    raise ValueError (' Task_done () called Too many times ')                Self.all_tasks_done.notify_all ()            self.unfinished_tasks = Unfinished      #否则未完成任务数量-1        finally:            self.all_tasks_done.release ()           #最后释放锁

This method determines whether the task of one thread in the queue is complete, first obtains the lock through the All_tasks_done object, notifies if it is, and finally releases the lock.


(4) Join ()

The source code is as follows:

def join (self):     self.all_tasks_done.acquire ()    try:        while self.unfinished_tasks:        #如果有未完成的任务, Call the Wait () method to wait for            self.all_tasks_done.wait ()    finally:        self.all_tasks_done.release ()

The blocking method, when there are unfinished processes in the queue, calls the join method to block until they are complete.


Other methods are relatively simple, but also relatively good understanding, interested can go to see the source code in queue.py, to note that any get queue status (empty (), qsize (), etc.), or modify the contents of the queue (Get,put, etc.) must hold the mutex mutex.

(b) Simple examples

Implement a thread that constantly generates a random number into a queue

Implement a thread that constantly takes out an odd number from the top of the queue

Implement another thread to continuously remove even numbers from the queue above

Import random,threading,timefrom Queue Import queueis_product = Trueclass Producer (threading. Thread): "" "Production Data" "" Def __init__ (self, T_name, queue): Threading.                 thread.__init__ (self,name=t_name) Self.data=queue def run (self): while 1:if self.data.full (): Global Is_product is_product = False else:if self.data.qsize () &                         lt;= 7: #队列长度小于等于7时添加元素 is_product = True for I in range (2): #每次向队列里添加两个元素 Randomnum=random.randint (1,99) print "%s:%s is producing%d to the queue!"% (time.c                        Time (), Self.getname (), Randomnum) self.data.put (randomnum,false) #将数据依次存入队列                    Time.sleep (1) print "Deque length is%s"%self.data.qsize () Else: If Is_product:for I in range (2): # Randomnum = Random.randint (1, total) print "%s:%s is producing%d to the queue!"% (Time.ctime (), Self.get Name (), Randomnum) Self.data.put (randomnum,false) # stores the data sequentially into the queue time.                        Sleep (1) print "Deque length is%s"% self.data.qsize () Else: Pass print "%s:%s finished!"% (Time.ctime (), Self.getname ()) #Consumer ThreadClass Consumer_even (Threadi Ng. Thread): Def __init__ (self,t_name,queue): Threading.  thread.__init__ (self,name=t_name) Self.data=queue def run (self): while 1:if self.data.qsize ()                    > 7: #队列长度大于7时开始取元素 val_even = Self.data.get (False) if val_even%2==0: Print "%s:%s is consuming. %d in the queue is consumed! "% (Time.ctime (), Self.getname (), Val_even) Time.sleep (2) Els        E:self.data.put (Val_even)            Time.sleep (2) print "Deque length is%s"% self.data.qsize () Else:pa SS Class Consumer_odd (threading. Thread): Def __init__ (self,t_name,queue): Threading. Thread.__init__ (self, name=t_name) Self.data=queue def run (self): while 1:if self.data.qsize ( ) > 7:val_odd = Self.data.get (False) if Val_odd%2!=0:print "%s:%s I S consuming. %d in the queue is consumed! "% (Time.ctime (), Self.getname (), val_odd) Time.sleep (2) El  Se:self.data.put (val_odd) time.sleep (2) print "Deque length is%s"% Self.data.qsize () Else:pass #Main threaddef Main (): Queue = Queue (a) producer = produce    R (' Pro. ', queue) Consumer_even = Consumer_even (' Con_even. ', queue) consumer_odd = Consumer_odd (' con_odd. ', queue) Producer.start () Consumer_even.starT () Consumer_odd.start () Producer.join () Consumer_even.join () Consumer_odd.join () if __name__ = = ' __main__ ': Main ()
  • Contact Us

    The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

    If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

    A Free Trial That Lets You Build Big!

    Start building with 50+ products and up to 12 months usage for Elastic Compute Service

    • Sales Support

      1 on 1 presale consultation

    • After-Sales Support

      24/7 Technical Support 6 Free Tickets per Quarter Faster Response

    • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.