The example in this article describes how Python puts back a large page download when using Scrapy to collect data. Share to everyone for your reference. The specific analysis is as follows:
Add the following code to Settings.py,myproject for your project name
Copy the Code code as follows:
downloader_httpclientfactory = ' Myproject.downloader.LimitSizeHTTPClientFactory '
Custom restricted modules for downloading oversized pages
Copy the Code code as follows:
Max_response_size = 1048576 # 1Mb
From scrapy.core.downloader.webclient import Scrapyhttpclientfactory, Scrapyhttppagegetter
Class Limitsizepagegetter (Scrapyhttppagegetter):
def handleheader (self, Key, value):
Scrapyhttppagegetter.handleheader (self, key, value)
If key.lower () = = ' Content-length ' and int (value) > max_response_size:
Self.connectionlost (' oversized ')
Class Limitsizehttpclientfactory (Scrapyhttpclientfactory):
protocol = Limitsizepagegetter
Hopefully this article will help you with Python programming.