Understanding producer Consumer models and application examples in Python programming

What is the producer consumer model

At work, you might encounter a situation where a module is responsible for generating data that is handled by another module (the module here is generalized, which can be classes, functions, threads, processes, and so on). The module that produces the data is visually called the producer, and the module that processes the data is called the consumer. Between producers and consumers in the addition of a buffer zone, our image is called the warehouse, the producer is responsible for the warehouse into the goods, and consumers are responsible for the goods from the warehouse, which constitutes a producer of consumer models. The structure diagram is as follows:

Advantages of the producer consumer model:

1. Decoupling

Assume that producers and consumers are two classes respectively. If a producer calls a method of the consumer directly, the producer will have a dependency on the consumer (that is, coupling). In the future, if the consumer's code changes, it may affect the producers. If both are dependent on a buffer, the two are not directly dependent, and the coupling is correspondingly reduced.

For example, we go to the post office to deliver a letter, and if you don't use a mailbox (that is, a buffer), you have to give it to the postman directly. Some students say, it is not easy to give the postman directly? In fact, it is not easy, you have to know who is the postman, in order to give the letter to him (wearing the uniform, in case someone fake, it is miserable). This creates a reliance on you and the postman (the equivalent of strong coupling between producers and consumers). In case the postman substitutes for one day, you need to get to know each other (the equivalent of a consumer change that leads to a change in producer code). The mailbox is relatively fixed, and you rely on it at a lower cost (equivalent to weak coupling between buffers).

2. Support Concurrency

Since producers and consumers are two separate concurrent bodies, they are connected using buffers as bridges, producers can continue to produce the next data only by throwing data into the buffer, and consumers only need to take the data from the buffer, so that they do not block because of each other's processing speed.

In the example above, if we don't use a mailbox, we'll have to wait for the postman at the post office until he comes back, we'll give him the mail, we can't do anything during that time (that is, producers are blocked), or the postman has to ask the door from house to house, who wants to send a letter (equivalent to a consumer poll).

3. Support free and busy uneven

The buffer also has another benefit. If the speed of manufacturing data is fast and slow, the benefits of the buffer are reflected. When the data is manufactured fast, the consumer is too late to handle it, and the unhandled data can be temporarily present in the buffer. And so the producers slow down the production, consumers and then slowly dispose of.

For full reuse, let's take the example of sending a letter. Suppose the postman can only take 1000 letters at a time. In case of a Valentine's Day (or maybe Christmas) to send a greeting card, need to send more than 1000 letters, this time the mailbox this buffer will come in handy. The postman put the letter which was too late to be taken in the mailbox, and then took it next time.

Python Example:
The use of queues to implement a simple producer consumer model, producer generation time into the queue, consumers take time to print

Class Consumer (threading. Thread):  def __init__ (self, queue):    Threading. Thread.__init__ (self)    self._queue = Queue  def run (self): while    True:      msg = Self._queue.get ()      if Isinstance (msg, str) and MSG = = ' quit ': Break      print "I ' m a thread, and I received%s!!"% msg    print ' Bye byes! ' Def producer ():  queue = Queue.queue ()  worker = Consumer (queue)  Worker.start () # Turn on consumer thread  Start_ Time = Time.time () while  time.time ()-start_time < 5:    queue.put (' Something at%s '% time.time ()) Time    . Sleep (1)  queue.put (' Quit ')  worker.join () if __name__ = = ' __main__ ':  producer ()

Using multi-threading, when doing the crawler, the producer used to generate URL links, consumers to obtain URL data, in the queue with the help of the use of multithreading to speed up the crawler.

Import timeimport threadingimport queueimport urllib2class Consumer (threading. Thread): Def __init__ (self, queue): Threading. Thread.__init__ (self) self._queue = Queue def run (self): while true:content = Self._queue.get () print Co ntent if isinstance (content, str) and content = = ' quit ': Break response = urllib2.urlopen (content) PRI NT ' Bye byes! ' Def Producer (): urls = [' http://211.103.242.133:8080/Disease/Details.aspx?id=2258 ', ' Http://211.103.242.133:8080/D isease/details.aspx?id=2258 ', ' http://211.103.242.133:8080/Disease/Details.aspx?id=2258 ', ' HTTP// 211.103.242.133:8080/disease/details.aspx?id=2258 '] queue = Queue.queue () worker_threads = Build_worker_pool (Queue, 4 ) Start_time = Time.time () for the URL in urls:queue.put (URL) for worker in Worker_threads:queue.put (' Quit ') for W Orker in Worker_threads:worker.join () print ' done! Time taken: {} '. Format (Time.time ()-start_time) def build_worker_pool (queue, size): WorkERS = [] for _ in range (size): worker = Consumer (queue) Worker.start () workers.append (worker) return Workersif __name__ = = ' __main__ ': Producer ()

