Scrapy middleware Summary

1. Downloader middleware: You can customize the middleware and the medium price priority;

I. How to add downloader middleware? RewriteProcess_request,Process_response,Process_exceptionFunction;

Ii. Why downloader middleware? Rewrite the request or specify the download behavior. For example, whether to send a cookie, specify the cache mechanism, Retry Mechanism, and how to process the redirected data.

Iii. What are the built-in downloader middleware? About 13

1. Cookiesmiddleware: Whether to forwardWeb ServerSendCookie
2. Defaultheadersmiddleware: Set allRequestSet the header to the default mode.
3. Downloadtimeoutmiddleware: SetRequestOfTimeout
4. Httpauthmiddleware:SpiderOfRequestAuthorization
5. Httpcachemiddleware:Request & ResponseSet cache policies
6. Httpcompressionmiddleware:
7. Chunkedtransfermiddleware:
8. Httpproxymiddleware: To allRequestSetHTTPProxy
9. Redirectmiddleware: ProcessingRequestRedirection
10. Metarefreshmiddleware: AccordingMeta-Refresh HTML Tag processing redirection
11. Retrymiddleware: Retry upon failure Policy
12. Robotstxtmiddleware:Robots bans
13. Useragentmiddleware: SupportedUser AgentRewrite

 

B)SpiderMiddleware: Customizable middleware and priority;

IV.How to add a newSpiderMiddleware? Heavy LoadProcess_spider_input,Process_spider_output,Process_spider_exception,Process_start_requests

V.Built-inSpiderWhat are middleware Products? About5Type

1. Depthmiddleware: Sets the priority of capture depth and depth;
2. Httperrormiddleware: Set the range of successful Codes
3. Offsitemiddleware: Primary domain filtering
4. Referermiddleware:
5. Urllengthmiddleware: VALIDURLLength