Example of using Python threads to solve the problem of producer consumption _python

We'll use Python threads to solve the producer-consumer problem in Python. The problem is not so difficult as they say in school.

If you have a knowledge of producer-consumer issues, it makes more sense to read this blog.

Why pay attention to producer-consumer issues:

• can help you to better understand concurrency and different concepts of concurrency.
• In the implementation of information queues, the concept of producer-consumer issues is used to some extent, and you are bound to use message queues at some point.

When we are using threads, you can learn the following threading concepts:

• Condition: The condition in the thread.
• Wait (): The Wait () that is available in the conditional instance.
• Notify (): Notify () available in a conditional instance.

I assume you already have these basic concepts: threads, race conditions, and how to resolve static conditions (for example, using lock). Otherwise, you suggest you go to see my last article Basics of Threads.

Quoting Wikipedia:

The producer's job is to produce a piece of data, put it in the buffer, and loop. At the same time, consumers are consuming the data (such as removing them from the buffer), one at a time.

The key word here is "at the same time." So producers and consumers are running concurrently, and we need to do a thread separation between producers and consumers.

From threading Import Thread
 
class Producerthread (thread):
  def run (self):
    Pass
 
class Consumerthread ( Thread):
  def run (self):
    Pass

To quote Wikipedia again:

This is a process that describes two shared fixed-size buffer queues, i.e. producers and consumers.

Suppose we have a global variable that can be modified by the producer and consumer threads. The producer generates the data and adds it to the queue. The consumer consumes the data (for example, moving it out).

Queue = []

At the beginning, we will not set a fixed size condition and join in the actual runtime (refer to the example below).

Start with a bug program:

From threading import Thread, lock
Import time
import random
 
queue = []
lock = Lock ()
 
class Producert Hread (Thread):
  def run (self):
    nums = Range (5) #Will Create the list [0, 1, 2, 3, 4]
    global queue while
    Tru E:
      num = Random.choice (nums) #Selects a random number from list [0, 1, 2, 3, 4]
      lock.acquire ()
      queue.append (num)
      Print "produced", Num
      lock.release ()
      time.sleep (Random.random ())
 
class Consumerthread (Thread):
  def run (self):
    global Queue while
    True:
      lock.acquire ()
      if not queue:
        print Queue, but consumer'll try to consume "
      num = queue.pop (0)
      print" Consumed ", num
      lock.release ()
      Time.sleep (Random.random ())
 
Producerthread (). Start ()
Consumerthread (). Start ()

Run several times and pay attention to the results. If the program does not end automatically after the indexerror exception, run with Ctrl+z.

Sample output:

Produced 3
consumed 3
produced 4
consumed 4
produced 1
consumed 1 Nothing
in queue, but consumer Would try to consume
Exception in thread Thread-2:
traceback (most recent called last):
 File "/usr/lib/python2.7 /threading.py ", line 551, in __bootstrap_inner
  self.run ()
 File" producer_consumer.py ", line, in run
  num = Queue.pop (0)
indexerror:pop from empty list

Explain:

• We started with a producer thread (hereinafter called a producer) and a consumer thread (hereinafter called the consumer).
• Producers keep adding (data) to queues, and consumers are constantly consuming them.
• Since the queue is a shared variable, we put it in a lock program block to prevent the occurrence of a race condition.
• At a certain point in time, the consumer consumes everything and the producer is still hanging (sleep). The consumer tried to continue consuming, but the queue was empty and a indexerror exception occurred.
• During each run, you will see the print statement output "Nothing in queue, but consumer would try to consume" before a Indexerror exception occurs, which is why you are making a mistake.

We use this implementation as error behavior (wrong behavior).

What is the right behavior?

When there is no data in the queue, the consumer should stop running and wait, rather than continue trying to consume. And when producers add data to the queue, there should be a channel to tell (notify) consumers. Consumers can then consume from the queue again, and Indexerror no longer appear.

About conditions

A condition (condition) allows one or more threads to enter a wait until it is notify by another thread. Reference:? http://docs.python.org/2/library/threading.html#condition-objects

That's all we need. We want the consumer to wait while the queue is empty, only to recover after being notify by the producer. The producer only notify after adding data to the queue. Therefore, after the producer notify, you can ensure that the queue is not empty, so the consumer does not have an exception when consuming.

• Condition contains lock.
• Condition has a acquire () and release () method to invoke the corresponding method of internal lock.

The lock Acquire () and release () are called internally by the condition acquire () and release () methods. So we can replace the lock instance with the Condiction instance, but the lock behavior will not change.
Producers and consumers need to use the same condition instance to ensure that wait and notify work properly.

Rewrite consumer code:

From threading import Condition
 
Condition = Condition ()
 
class Consumerthread (Thread):
  def run (self):
    Global Queue
    while True:
      condition.acquire ()
      if not queue:
        print ' Nothing in queue ', consumer is waiting '
        condition.wait ()
        print "Producer added something to queue and notified the consumer"
      num = queue.pop (0) 
   print "consumed", num
      condition.release ()
      time.sleep (Random.random ())

Rewrite producer Code:

Class Producerthread (Thread):
  def run (self):
    nums = Range (5)
    global queue while
    True:
      Condition.acquire ()
      num = Random.choice (nums)
      queue.append (num)
      print "produced", Num
      Condition.notify () condition.release () Time.sleep () (
      random.random ())

Sample output:

Produced 3
consumed 3
produced 1
consumed 1
produced 4
consumed 4
produced 3
consumed 3 Nothing
in the queue, consumer is waiting
Produced 2
Producer added something to queue and notified the consumer
consumed 2 no
in queue, consume  R is waiting
produced 2
Producer added something to queue and notified the consumer
consumed 2
Nothing In queue, consumer are waiting
produced 3
Producer added something to queue and notified the consumer
Consu Med 3
produced 4
consumed 4
produced 1
consumed 1

Explain:

• For consumers, check to see if the queue is empty before consuming.
• If NULL, the Wait () method of the condition instance is invoked.
• The consumer enters wait () while releasing the lock held.
• It will not run unless it is notify.
• The producer can acquire this lock because it has been release by the consumer.
• When the condition Notify () method is invoked, the consumer is awakened, but the wake-up does not mean it can start running.
• Notify () does not release lock, after invoking notify (), lock is still held by the producer.
• The producer explicitly releases the lock by Condition.release ().
• The consumer starts running again, and now it can get the data in the queue without indexerror exceptions.

Increase the size limit for queues

Producers cannot continue to add data to a full queue.

It can be implemented in the following ways:

• Before data is added, the producer checks whether the queue is full.
• If not full, the producer can continue the normal process.
• If it is full, the producer must wait, calling the condition instance's waiting ().
• Consumers can run. Consumers consume queues and generate a free position.
• Then consumers notify producers.
• When the consumer releases lock, the consumer can acquire the lock and then add data to the queue.

The final procedure is as follows:

From threading import Thread, Condition import time import random queue = [] Max_num = Condition = Condition () CLA 
      SS Producerthread (Thread): def run (self): Nums = Range (5) Global queue while True:condition.acquire () If Len (queue) = = Max_num:print "Queue full, producer is Waiting" condition.wait () print "S Pace in queue, Consumer notified the producer "num = Random.choice (nums) queue.append (num) print" Produc Ed ", Num condition.notify () condition.release () Time.sleep (Random.random ()) class Consumerthread (Threa  d): def run (self): Global queue while True:condition.acquire () if not queue:print ' nothing  In queue, consumer are Waiting "condition.wait () print" Producer added something to queue and notified the Consumer "num = Queue.pop (0) print" consumed ", num condition.notify () condition.release () t Ime.sleep (Random.random () Producerthread (). Start () Consumerthread (). Start ()

 

Sample output:

Produced 0
consumed 0
produced 0
produced 4
consumed 0
consumed 4 Nothing into
queue, consumer is W Aiting
produced 4
Producer added something to queue and notified the consumer
consumed 4
produced 3
   
    produced 2
consumed 3

   

Update:
Many netizens suggest that I use queue under lock and condition instead of using list. I agree with this approach, but my goal is to show how Condition,wait () and notify () work, so use the list.

The following queue is used to update the code.

The queue encapsulates the behavior of condition, such as Wait (), notify (), acquire ().

Now it's a good chance to read the queue document (http://docs.python.org/2/library/queue.html).

Update program:

From threading import Thread
Import time
import random from
queue import \ queue
 
= Queue
 
Class Producerthread (Thread):
  def run (self):
    nums = Range (5)
    global queue while
    True:
      num = Random.choice (nums)
      queue.put (num)
      print "produced", Num
      time.sleep (random.random ())
 
class Consumerthread (Thread):
  def run (self):
    global Queue while
    True:
      num = queue.get ()
      Queue.task_done ()
      print "Consumed", num
      time.sleep (random.random ())
 
Producerthread (). Start ()
Consumerthread (). Start ()

Explain:

•     Use the queue instance (the next queue) instead of the location where the list was originally used.
•     This queue has a condition, it has its own lock. If you use queue, you don't have to worry about condition and lock.
•     The Put method of the producer call queue to insert the data.
•     put () has a logic to get lock before inserting data.
•     Also, put () checks whether the queue is full. If it is full, it calls the wait () internally, and the producer starts waiting.
•     Consumers use the Get method. The
•     get () gets lock before it is removed from the queue.
•     Get () checks whether the queue is empty, and if it is empty, the consumer enters the wait state.
•     get () and put () have the appropriate notify (). Now to see the queue's source code bar.