Understand producer and consumer models and use cases in Python Programming

The producer consumer model is generally used to reflect the multi-thread concurrency of a program. although Python's multithreading is controlled by GIL, it can still be used to build a queue to reflect the idea of the model, here we will come to understand the producer consumer model and examples of using it in Python programming: What is a producer-consumer model?

At work, you may encounter a situation where a module is responsible for generating data and the data is processed by another module (the module here is in a broad sense, can be a class, function, thread, process, etc ). The module of production data is known as a producer, while the module for data processing is called a consumer. Adding a buffer zone between the producer and the consumer is called a warehouse. the producer is responsible for importing goods to the warehouse, and the consumer is responsible for taking goods from the warehouse, this constitutes the producer and consumer model. The structure is as follows:

Advantages of the producer consumer model:

1. decoupling

Assume that the producer and consumer are two classes. If the producer directly calls a method of the consumer, the producer will depend on the consumer (that is, coupling ). In the future, if the consumer's code changes, it may affect the producer. If both of them depend on a buffer zone, the coupling between the two is reduced accordingly.

For example, if you do not use a mailbox (that is, a buffer zone), you must send the mail directly to the postman. Some people may say that it is quite easy to directly give the postman? In fact, it is not easy. you must know who is a postman to give the mail to him (just by the uniform on your body, in case someone is fake, it will be miserable ). This produces dependencies with you and The Postman (equivalent to strong coupling between producers and consumers ). In case the postman changes people one day, you need to know it again (equivalent to modifying the producer code due to consumer changes ). The mailbox is relatively fixed, and the cost of relying on it is relatively low (equivalent to weak coupling with the buffer zone ).

2. support for concurrency

Since the producer and consumer are two independent concurrent bodies, they are connected using a buffer zone as a bridge. the producer can continue to produce the next data by dropping data in the buffer zone, consumers only need to get data from the buffer zone, so that they will not block each other because of their processing speed.

In the following example, if we do not use a mail box, we have to wait for the postmaster at the post office until he returns. we will give the mail to him, during this period, we can't do anything (that is, the producer is blocked), or the postman has to ask door-to-door who wants to send messages (equivalent to consumer polling ).

3. support for idle and busy periods

The buffer has another benefit. If the speed of data manufacturing is fast and slow, the benefits of the buffer zone will be reflected. When data is created quickly, the consumer cannot process the data. unprocessed data can be temporarily stored in the buffer zone. When the producer's manufacturing speed slows down, the consumer will slowly process it.

In order to fully reuse the information, let's take the mail example as an example. Assume that the postman can only take away 1000 messages at a time. If a greeting card is sent on Valentine's Day (or may be Christmas), more than 1000 emails need to be sent. then the buffer zone of the mailbox will be used. The Postman saves the delayed emails in the inbox and takes them again when they arrive.

Python example:
A simple producer-consumer model is implemented using queues. the producer generation time is put into the queue, and the consumer retrieval time is printed.

Class Consumer (threading. thread): def _ init _ (self, queue): threading. thread. _ init _ (self) self. _ queue = queue def run (self): while True: msg = self. _ queue. get () if isinstance (msg, str) and msg = 'Quit': break print "I'm a thread, and I got Ed % s !! "% Msg print 'Bye byes! 'Def producer (): queue = Queue. queue () worker = Consumer (queue) worker. start () # start consumer thread start_time = time. time () while time. time ()-start_time <5: queue. put ('something at % s' % time. time () time. sleep (1) queue. put ('quit') worker. join () if _ name _ = '_ main _': producer ()

With multithreading, the producer generates url links when crawling. the consumer obtains url data. with the help of the queue, multithreading can be used to speed up crawling.

import timeimport threadingimport Queueimport urllib2class Consumer(threading.Thread):  def __init__(self, queue):    threading.Thread.__init__(self)    self._queue = queue  def run(self):    while True:      content = self._queue.get()      print content      if isinstance(content, str) and content == 'quit':        break      response = urllib2.urlopen(content)    print 'Bye byes!'def Producer():  urls = [    'http://211.103.242.133:8080/Disease/Details.aspx?id=2258',    'http://211.103.242.133:8080/Disease/Details.aspx?id=2258',    'http://211.103.242.133:8080/Disease/Details.aspx?id=2258',    'http://211.103.242.133:8080/Disease/Details.aspx?id=2258'  ]  queue = Queue.Queue()  worker_threads = build_worker_pool(queue, 4)  start_time = time.time()  for url in urls:    queue.put(url)  for worker in worker_threads:    queue.put('quit')  for worker in worker_threads:    worker.join()  print 'Done! Time taken: {}'.format(time.time() - start_time)def build_worker_pool(queue, size):  workers = []  for _ in range(size):    worker = Consumer(queue)    worker.start()    workers.append(worker)  return workersif __name__ == '__main__':  Producer()