Using Python multi-threading to implement producer-consumer patterns crawl the image of the bucket graph network

What is the producer consumer model

Some modules are responsible for production data, which is handled by other modules (the modules here may be: functions, threads, processes, etc.). The module that produces the data is called the producer, and the module that processes the data is called the consumer. The buffer between producer and consumer is called a warehouse. Producers are responsible for transporting goods to the warehouse, and consumers are responsible for taking the goods out of the warehouse, which constitutes the producer's consumer model.

Advantages of the producer consumer model

• Decoupling
  Assume that the producer and consumer are two threads respectively. If a producer calls a method of the consumer directly, the producer will have a dependency on the consumer (that is, coupling). If the code of the future consumer changes, it may affect the code of the producer. If both are dependent on a buffer, the two are not directly dependent, and the coupling is correspondingly reduced.

• Concurrent
  Since producers and consumers are two separate concurrent bodies that communicate with each other by using buffers, producers can continue to produce the next data only by throwing data into the buffer, and consumers simply need to take the data from the buffer so that they do not block because of each other's processing speed.

• Support for non-uniform free and busy
  When the producer makes the data fast, the consumer is too late to deal with it, the unhandled data can be temporarily present in the buffer and slowly disposed of. Not because of consumer performance caused by data loss or affect producer production.

Instance

#!/usr/bin/python#-*-coding:utf-8-*-# @Time: 2017/12/4 16:29# @Author: Yulei lan# @Email: [Email protect ed]# @File: urls.py# @Software: Pycharmimport requestsimport osfrom urllib.request import urlretrievefrom BS4 import B Eautifulsoupimport threadingbase_page_url = ' http://www.doutula.com/photo/list/?page= ' page_url_list = [] # List of all pagination face_url_list = [] # URL List of all emoticons Glock = threading.  Lock () def get_page_url (): For I in range (1, 2): url = base_page_url + str (i) page_url_list.append (URL) def Procuder (): "" "producer keeps producing all downloadable Img_url addresses" "While Len (page_url_list)! = 0: # Cannot use for loop g        Lock.acquire () Page_url = Page_url_list.pop () glock.release () response = Requests.get (Page_url) Text = Response.text Soup = beautifulsoup (text, ' lxml ') img_list = Soup.find_all (' img ', attrs={' class ': ' I Mg-responsive lazy Image_dta '}) Glock.acquire () for img in Img_list:img_url = img[' data-original '] if not img_url.startswith (' http: '): Img_url = ' http:{} '. Format (Img_url) Face_url_list.append (Img_url) glock.release () def customer (): "" "Consumer" "" While True:if Len (F ace_url_list) = = 0:continue Else:img_url = Face_url_list.pop () tmp_list = Img_ur L.split ('/') filename = Tmp_list.pop () Path_file = Os.path.join (' images ', filename) urlre Trieve (Img_url, Filename=path_file) def main (): For I in range (3): th = Threading. Thread (Target=procuder) Th.start () for I in range (5): th = Threading. Thread (Target=customer) Th.start () if __name__ = = ' __main__ ': Get_page_url () Main ()

Using Python multithreading to implement producer-consumer patterns crawl Bucket graph Web emoticons