Example of using a Python thread to solve the producer consumption problem.

Example of using a Python thread to solve the producer consumption problem.

We will use the Python thread to solve the producer-consumer problem in Python. This problem is not as difficult as they mentioned at school.

If you have an understanding of producer-consumer issues, reading this blog will be more meaningful.

Why are you concerned about producer-consumer issues:

• It helps you better understand concurrency and concurrency with different concepts.
• In the implementation of information queue, the concept of producer-consumer is used to a certain extent, and you will inevitably use message queue in some cases.

When using a thread, you can learn the following thread concepts:

• Condition: Conditions in the thread.
• Wait (): wait () available in the condition instance ().
• Y (): available notify () in the condition instance ().

I suppose you already have these basic concepts: thread, race condition, and how to solve static conditions (such as using lock ). Otherwise, you are advised to go to my previous article basics of Threads.

Reference Wikipedia:

The producer's job is to generate a piece of data and put it in the buffer. At the same time, the consumer is consuming the data (such as removing them from the buffer), each piece.

The keyword here is "at the same time ". Therefore, the producer and consumer run concurrently. We need to separate the producer and consumer threads.
 

from threading import Thread class ProducerThread(Thread):  def run(self):    pass class ConsumerThread(Thread):  def run(self):    pass

Reference Wikipedia again:

This describes two processes that share a fixed-size buffer queue, namely, producers and consumers.

Suppose we have a global variable that can be modified by the producer and consumer threads. The producer generates data and adds it to the queue. The consumer consumes the data (for example, removing it ).

Queue = []

At the beginning, we will not set a fixed size condition, but add it to the actual running (referring to the following example ).

Programs with bugs at the beginning:

from threading import Thread, Lockimport timeimport random queue = []lock = Lock() class ProducerThread(Thread):  def run(self):    nums = range(5) #Will create the list [0, 1, 2, 3, 4]    global queue    while True:      num = random.choice(nums) #Selects a random number from list [0, 1, 2, 3, 4]      lock.acquire()      queue.append(num)      print "Produced", num      lock.release()      time.sleep(random.random()) class ConsumerThread(Thread):  def run(self):    global queue    while True:      lock.acquire()      if not queue:        print "Nothing in queue, but consumer will try to consume"      num = queue.pop(0)      print "Consumed", num      lock.release()      time.sleep(random.random()) ProducerThread().start()ConsumerThread().start()

Run it several times and pay attention to the results. If the program does not automatically end after the IndexError exception, use Ctrl + Z to end the operation.

Sample output:
 

Produced 3Consumed 3Produced 4Consumed 4Produced 1Consumed 1Nothing in queue, but consumer will try to consumeException in thread Thread-2:Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner  self.run() File "producer_consumer.py", line 31, in run  num = queue.pop(0)IndexError: pop from empty list

Explanation:

• We started a producer thread (hereinafter referred to as the producer) and a consumer thread (hereinafter referred to as the consumer ).
• The producer continuously adds (data) to the queue, while the consumer continuously consumes.
• Because the queue is a shared variable, we put it in the lock block to prevent race conditions.
• At a certain point in time, the consumer consumes everything and the producer is still suspended (sleep ). The consumer tries to continue consumption, but the queue is empty at this time, and an IndexError exception occurs.
• During each running process, before an IndexError exception occurs, you will see the print statement output "Nothing in queue, but consumer will try to consume", which is the cause of your error.

We use this implementation as the error behavior (wrong behavior ).

What is correct?

When there is no data in the queue, the consumer should stop running and wait (wait), rather than continue to try to consume. When the producer adds data to the queue, there should be a channel to notify the consumer. Then, the consumer can consume data from the queue again, but the IndexError does not appear.

About Conditions

Condition allows one or more threads to enter wait until it is replaced by another thread. Reference :? Http://docs.python.org/2/library/threading.html#condition-objects

This is what we need. We hope that the consumer can only restore the wait when the queue is empty after being notify by the producer. The producer only performs notify after adding data to the queue. Therefore, after producer y, you can ensure that the queue is not empty, so there is no exception during consumer consumption.

• Condition contains lock.
• Condition has the acquire () and release () Methods to call the corresponding internal lock methods.

The acquire () and release () Methods of condition call lock's acquire () and release () internally (). Therefore, we can replace the lock instance with a condiction instance, but the lock action will not change.
The producer and consumer must use the same condition instance to ensure that wait and consumer y work normally.

Rewrite the Consumer Code:
 

from threading import Condition condition = Condition() class ConsumerThread(Thread):  def run(self):    global queue    while True:      condition.acquire()      if not queue:        print "Nothing in queue, consumer is waiting"        condition.wait()        print "Producer added something to queue and notified the consumer"      num = queue.pop(0)      print "Consumed", num      condition.release()      time.sleep(random.random())

Rewrite the producer code:
 

class ProducerThread(Thread):  def run(self):    nums = range(5)    global queue    while True:      condition.acquire()      num = random.choice(nums)      queue.append(num)      print "Produced", num      condition.notify()      condition.release()      time.sleep(random.random())

Sample output:
 

Produced 3Consumed 3Produced 1Consumed 1Produced 4Consumed 4Produced 3Consumed 3Nothing in queue, consumer is waitingProduced 2Producer added something to queue and notified the consumerConsumed 2Nothing in queue, consumer is waitingProduced 2Producer added something to queue and notified the consumerConsumed 2Nothing in queue, consumer is waitingProduced 3Producer added something to queue and notified the consumerConsumed 3Produced 4Consumed 4Produced 1Consumed 1

Explanation:

• For consumers, check whether the queue is empty before consumption.
• If it is null, call the wait () method of the condition instance.
• The consumer enters wait () and releases the lock.
• It does not run unless it is specified by Y.
• The producer can use the acquire lock because it has been release by the consumer.
• After the condition notify () method is called, the consumer is awakened, But waking does not mean it can start running.
• Notify () does not release the lock. After notify () is called, the lock is still held by the producer.
• The producer uses condition. release () to explicitly release the lock.
• The consumer starts running again. Now it can get the data in the queue without the IndexError exception.

Increase the queue size limit

The producer cannot add data to a full queue.

It can be implemented in the following ways:

• Before adding data, the producer checks whether the queue is full.
• If it is not full, the producer can continue the normal process.
• If it is full, the producer must wait and call the wait () of the condition instance ().
• Consumers can run. The consumer consumes a queue and generates a free location.
• Then the consumer producer y.
• When the consumer releases the lock, the consumer can acquire the lock and then add data to the queue.

The final procedure is as follows:

from threading import Thread, Conditionimport timeimport random queue = []MAX_NUM = 10condition = Condition() class ProducerThread(Thread):  def run(self):    nums = range(5)    global queue    while True:      condition.acquire()      if len(queue) == MAX_NUM:        print "Queue full, producer is waiting"        condition.wait()        print "Space in queue, Consumer notified the producer"      num = random.choice(nums)      queue.append(num)      print "Produced", num      condition.notify()      condition.release()      time.sleep(random.random()) class ConsumerThread(Thread):  def run(self):    global queue    while True:      condition.acquire()      if not queue:        print "Nothing in queue, consumer is waiting"        condition.wait()        print "Producer added something to queue and notified the consumer"      num = queue.pop(0)      print "Consumed", num      condition.notify()      condition.release()      time.sleep(random.random()) ProducerThread().start()ConsumerThread().start()

Sample output:
 

Produced 0Consumed 0Produced 0Produced 4Consumed 0Consumed 4Nothing in queue, consumer is waitingProduced 4Producer added something to queue and notified the consumerConsumed 4Produced 3Produced 2Consumed 3

Update:
Many netizens suggest using Queue in lock and condition instead of list. I agree with this approach, but my purpose is to demonstrate how Condition, wait (), and Policy () work, so the list is used.

The following uses Queue to update the code.

Queue encapsulates Condition behaviors, such as wait (), Policy (), and acquire ().

Now is a good opportunity to read the Queue documentation (http://docs.python.org/2/library/queue.html ).

Update Program:

from threading import Threadimport timeimport randomfrom Queue import Queue queue = Queue(10) class ProducerThread(Thread):  def run(self):    nums = range(5)    global queue    while True:      num = random.choice(nums)      queue.put(num)      print "Produced", num      time.sleep(random.random()) class ConsumerThread(Thread):  def run(self):    global queue    while True:      num = queue.get()      queue.task_done()      print "Consumed", num      time.sleep(random.random()) ProducerThread().start()ConsumerThread().start()

Explanation:

• In the original position where the list is used, the Queue instance (Queue) is used instead ).
• This queue has a condition, which has its own lock. If you use Queue, you do not need to worry about condition and lock.
• The producer calls the put Method of the queue to insert data.
• Put () has a logic to get the lock before inserting data.
• Put () also checks whether the queue is full. If it is full, it calls wait () internally and the producer starts to wait.
• The consumer uses the get method.
• Get () gets the lock before removing data from the queue.
• Get () checks whether the queue is empty. If it is empty, the consumer enters the waiting state.
• Get () and put () both have appropriate y (). Now let's look at the source code of Queue.