Example of using Python threads to solve producer consumption problems

We will use the Python thread to solve the producer-consumer problem in Python. The problem is not as difficult as they say in school.

If you have an understanding of producer-consumer issues, it makes more sense to read this blog post.

Why care about producer-consumer issues:

• can help you to better understand concurrency and different concepts of concurrency.
• In the implementation of the information queue, the concept of producer-consumer issues is used to some extent, and you will inevitably use Message Queuing.

When we are using threads, you can learn the following threading concepts:

• Condition: A condition in a thread.
• Wait (): Wait () is available in the condition instance.
• Notify (): Notify () available in the condition instance.

I assume that you already have these basic concepts: threads, race conditions, and how to resolve static conditions (such as using lock). Otherwise, you suggest you go and see my last article Basics of Threads.

cite Wikipedia:

The job of a producer is to produce a piece of data, put it in buffer, and so loop. At the same time, consumers consume this data (such as removing them from buffer), one at a time.

The key word here is "at the same time". So producers and consumers are running concurrently, and we need to do thread separation for producers and consumers.

From threading Import thread class Producerthread (thread):  def run:    Pass class Consumerthread (thread):  def run (self):    Pass

Quote Wikipedia again:

This describes the two processes that share a fixed-size buffer queue, that is, producers and consumers.

Suppose we have a global variable that can be modified by the producer and consumer threads. The producer generates the data and adds it to the queue. Consumers consume this data (such as moving it out).

Queue = []

At the beginning, we will not set a fixed size condition and add it to the actual runtime (refer to the example below).

Start with the bug program:

From threading import Thread, Lockimport timeimport random queue = []lock = Lock () class Producerthread (Thread):  def R Un (self):    nums = Range (5) #Will Create the list [0, 1, 2, 3, 4]    global queue while    True:      num = Random.choi  CE (nums) #Selects a random number from list [0, 1, 2, 3, 4]      lock.acquire ()      queue.append (num)      print "produced", Num      lock.release ()      time.sleep (Random.random ()) class Consumerthread (Thread):  def run (self):    Global Queue    while True:      lock.acquire ()      if not queue:        print ' Nothing ' in the queue, but consumer'll try T o consume "      num = queue.pop (0)      print" Consumed ", num      lock.release ()      time.sleep (Random.random ()) Producerthread (). Start () Consumerthread (). Start ()

Run a few times and pay attention to the results. If the program does not end automatically after the indexerror exception, run with CTRL + Z.

Sample output:

Produced 3Consumed 3Produced 4Consumed 4Produced 1Consumed 1Nothing in queue, but consumer'll try to consumeexception in Thread Thread-2:traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner  Self.run () File "producer_consumer.py", line-to-run  num = queue.pop (0) Indexerror:pop from empty list

Explain:

• We started a producer thread (hereinafter called producer) and a consumer thread (the consumer).
• Producers keep adding (data) to the queue, and consumers are constantly consuming it.
• Since the queue is a shared variable, we put it inside the lock block in case of a race condition.
• At a certain point in time, the consumer consumes everything and the producer hangs up (sleep). The consumer tries to continue consuming, but at this point the queue is empty and a Indexerror exception occurs.
• In each run, you will see the print statement output "Nothing in the queue, but consumer'll try to consume" before the indexerror exception occurs, which is why you went wrong.

We take this implementation as a false behavior (wrong behavior).

What is the right behavior?

When there is no data in the queue, the consumer should stop running and wait (wait) instead of trying to consume. And when the producer joins the data in the queue, there should be a channel to tell the (notify) consumer. The consumer can then consume from the queue again, and Indexerror no longer appears.

About conditions

A condition (condition) allows one or more threads to enter wait until they are notify by another thread. Reference:? http://docs.python.org/2/library/threading.html#condition-objects

That's all we need. We want the consumer to wait while the queue is empty, only to be restored after the producer notify. The producer notify only after adding data to the queue. Therefore, after the producer notify, you can ensure that the queue is non-empty, so the consumer consumes without exception.

• Condition contains lock.
• The condition has the acquire () and release () methods to invoke the corresponding method of the internal lock.

The Acquire () and release () of lock are called internally by the Acquire () and release () methods of condition. So we can replace the lock instance with a Condiction instance, but the behavior of lock does not change.
Producers and consumers need to use the same condition instance to ensure that wait and notify work properly.

Rewrite consumer code:

From threading import Condition Condition = Condition () class Consumerthread (Thread):  def run:    Global Queue    while True:      condition.acquire ()      if not queue:        print "Nothing in queue, consumer is waiting"        Condition.wait ()        print "Producer added something to queue and notified the consumer"      num = queue.pop (0)      Print "Consumed", num      condition.release ()      time.sleep (Random.random ())

Rewrite producer Code:

Class Producerthread (Thread):  def run (self):    nums = Range (5)    global queue while    True:      Condition.acquire ()      num = Random.choice (nums)      queue.append (num)      print "produced", Num      Condition.notify ()      condition.release ()      time.sleep (Random.random ())

Sample output:

Produced 3Consumed 3Produced 1Consumed 1Produced 4Consumed 4Produced 3Consumed 3Nothing in queue, consumer is Waitingprodu CED 2Producer added something to queue and notified the consumerconsumed 2Nothing in queue, consumer is waitingproduced 2P Roducer added something to queue and notified the consumerconsumed 2Nothing in queue, consumer is waitingproduced 3Produce R added something to queue and notified the consumerconsumed 3Produced 4Consumed 4Produced 1Consumed 1

Explain:

• For consumers, check whether the queue is empty before consuming.
• If empty, the Wait () method of the condition instance is called.
• The consumer enters wait () and releases the lock it holds.
• It will not run unless it is notify.
• The producer can acquire this lock because it has been release by the consumer.
• When the condition Notify () method is called, the consumer is awakened, but waking does not mean that it can start running.
• Notify () does not release lock, and after calling notify (), lock is still held by the producer.
• The producer explicitly releases lock through Condition.release ().
• The consumer starts running again, and now it can get the data in the queue without indexerror exceptions.

Increase the size limit for a queue

The producer cannot continue to add data to a full queue.

It can be implemented in the following ways:

• Before data is added, the producer checks whether the queue is full.
• If not full, the producer can continue the normal process.
• If it is full, the producer must wait to invoke the wait () of the condition instance.
• Consumers can run. The consumer consumes the queue and generates a spare position.
• Then consumers notify producers.
• When the consumer releases lock, the consumer can acquire the lock and then add data to the queue.

The final program is as follows:

From threading import Thread, Conditionimport timeimport random queue = []max_num = 10condition = Condition () class Produc Erthread (Thread): def run (self): Nums = Range (5) Global queue while True:condition.acquire () If Len (q Ueue) = = Max_num:print "Queue full, producer are Waiting" condition.wait () print "Space in Queue, Con Sumer notified the producer "num = Random.choice (nums) queue.append (num) print" produced ", Num Conditi On.notify () condition.release () Time.sleep (Random.random ()) class Consumerthread (Thread): def run (self): Glo        Bal queue while True:condition.acquire () if isn't Queue:print "Nothing in the queue, consumer is waiting"       Condition.wait () print "Producer added something to queue and notified the consumer" num = Queue.pop (0) Print "Consumed", num condition.notify () condition.release () Time.sleep (Random.random ()) Producerthre AD (). Start () ConsumerthreAD (). Start () 

Sample output:

Produced 0Consumed 0Produced 0Produced 4Consumed 0Consumed 4Nothing in queue, consumer is waitingproduced 4Producer added Something to queue and notified the consumerconsumed 4Produced 3Produced 2Consumed 3

Update:
Many netizens suggested that I use queue in place of lock and condition instead of using list. I agree with this approach, but my goal is to show how Condition,wait () and notify () work, so use the list.

Use queue to update the code below.

The queue encapsulates the behavior of condition, such as Wait (), notify (), acquire ().

Now is a good opportunity to read the queue's documentation (http://docs.python.org/2/library/queue.html).

Update program:

From threading import threadimport timeimport randomfrom queue Import Queue queue = Queue (Ten) class Producerthread (Thread) :  def run:    nums = Range (5)    global queue while    True:      num = Random.choice (nums)      queue.put (num)      Print "produced", Num      time.sleep (random.random ()) class Consumerthread (Thread):  def run (self):    Global Queue    while True:      num = queue.get ()      queue.task_done ()      print "Consumed", num      time.sleep ( Random.random ()) Producerthread (). Start () Consumerthread (). Start ()

Explain:

• In the original location using list, use queue instance (the queue) instead.
• This queue has a condition, and it has its own lock. If you use queue, you don't have to worry about condition and lock.
• The producer calls the queue's Put method to insert the data.
• Put () has a logic to get lock before inserting the data.
• Also, put () checks whether the queue is full. If it is full, it calls wait () internally, and the producer begins to wait.
• The consumer uses the Get method.
• Get () gets lock before the data is removed from the queue.
• Get () checks if the queue is empty and if it is empty, the consumer enters a wait state.
• Both get () and put () have the appropriate notify (). Now go and see the source of the queue.

