Chapter 1.3 Dynamic useragent of Scrapy

Prevent crawlers, starting with HTTP request header information, so useragent need to do dynamic settings

#-*-Coding:utf-8-*-"Created on April 21, 2017 user agent @author: DZM @param encryption Level ID: N: No security encryption, I: Weak security encryption, U: Strong secure encryption @param rendering engine: G Ecko, WebKit, Khtml, Presto, Trident, Tasman and other @see: http://www.cnblogs.com/junrong624/p/5533655.html prevention Crawler, Starting with the HTTP request header, user-agent in Python's urllib library is python-urllib, so user-agent can easily be suspected of "user_agents = [" mozilla/4.0 (Compatib Le MSIE 6.0; Windows NT 5.1; SV1; Acoobrowser;. NET CLR 1.1.4322;. NET CLR 2.0.50727) "," mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1;. NET CLR 2.0.50727; Media Center PC 5.0;. NET CLR 3.0.04506) "," mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; Aolbuild 4337.35; Windows NT 5.1;. NET CLR 1.1.4322,. NET CLR 2.0.50727) "," mozilla/5.0 (Windows; U MSIE 9.0; Windows NT 9.0; En-US) "," mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; trident/5.0;. NET CLR 3.5.30729;. NET CLR 3.0.30729;. NET CLR 2.0.50727; Media Center PC 6.0) "," mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; trident/4.0; WOW64; trident/4.0; SLCC2;. NET CLR 2.0.50727;. NET CLR 3.5.30729;. NET CLR 3.0.30729;. NET CLR 1.0.3705;. NET CLR 1.1.4322) "," MOZILLA/4. 0 (compatible; MSIE 7.0b; Windows NT 5.2;. NET CLR 1.1.4322;. NET CLR 2.0.50727; infopath.2;. NET CLR 3.0.04506.30) "," mozilla/5.0 (Windows; U Windows NT 5.1; ZH-CN) applewebkit/523.15 (khtml, like Gecko, safari/419.3) arora/0.3 (change:287 c9dfb30) "," mozilla/5.0 (X11; U Linux; En-US) applewebkit/527+ (khtml, like Gecko, safari/419.3) arora/0.6 "," mozilla/5.0 (Windows; U Windows NT 5.1; En-us; Rv:1.8.1.2pre) gecko/20070215 k-ninja/2.1.1 "," mozilla/5.0 (Windows; U Windows NT 5.1; ZH-CN; rv:1.9) gecko/20080705 firefox/3.0 kapiko/3.0 "," mozilla/5.0 (X11; Linux i686; U;) gecko/20070322 kazehakase/0.4.5 "," mozilla/5.0 (X11; U Linux i686; En-us; rv:1.9.0.8) Gecko fedora/1.9.0.8-1.fc10 kazehakase/0.5.6 "," mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/535.11 (khtml, like Gecko) chrome/17.0.963.56 safari/535.11 "," Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) applewebkit/535.20 (khtml, like Gecko) chrome/19.0.1036.7 safari/535.20 "," opera/9.80 (Macintos H Intel Mac OS X 10.6.8; U FR) presto/2.9.168 version/11.52 "," mozilla/5.0 (Windows NT 6.1;  WOW64) applewebkit/536.11 (khtml, like Gecko) chrome/20.0.1132.11 taobrowser/2.0 safari/536.11 "," mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.1 (khtml, like Gecko) chrome/21.0.1180.71 safari/537.1 lbbrowser "," mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; trident/5.0; SLCC2;. NET CLR 2.0.50727;. NET CLR 3.5.30729;. NET CLR 3.0.30729; Media Center PC 6.0;. net4.0c;. net4.0e; Lbbrowser) "," mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Qqdownload 732;. net4.0c;. net4.0e; Lbbrowser) "," mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/535.11 (khtml, like Gecko) chrome/17.0.963.84 safari/535.11 lbbrowser "," mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; trident/5.0; SLCC2;. NET CLR 2.0.50727;. NET CLR 3.5.30729;. NET CLR 3.0.30729; Media Center PC 6.0;. net4.0c;. net4.0e) "," mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; trident/5.0; SLCC2;. NET CLR 2.0.50727;. NET CLR 3.5.30729;. NET CLR 3.0.30729; Media Center PC 6.0;. net4.0c;. net4.0e; qqbrowser/7.0.3698.400) "," mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Qqdownload 732;. net4.0c;. net4.0e) "," mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; trident/4.0; SV1; Qqdownload 732;. net4.0c;. net4.0e; 360SE) "," mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Qqdownload 732;. net4.0c;. net4.0e) "," mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; trident/5.0; SLCC2;. NET CLR 2.0.50727;. NET CLR 3.5.30729;. NET CLR 3.0.30729; Media Center PC 6.0;. net4.0c;.
    net4.0e) "," mozilla/5.0 (Windows NT 5.1) applewebkit/537.1 (khtml, like Gecko) chrome/21.0.1180.89 safari/537.1 ", "mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.1 (khtml, like Gecko) chrome/21.0.1180.89 Safari/537.1 "," mozilla/5.0 (IPad; U CPU os 4_2_1 like Mac os X;  ZH-CN) applewebkit/533.17.9 (khtml, like Gecko) version/5.0.2 mobile/8c148 safari/6533.18.5 "," mozilla/5.0 (Windows NT 6.1; Win64; x64; Rv:2.0b13pre) gecko/20110307 Firefox/4.0b13pre "," mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) gecko/20100101 firefox/16.0 "," mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.11 (khtml, like Gecko) chrome/23.0.1271.64 safari/537.11 "," mozilla/5.0 (X11; U Linux x86_64; ZH-CN; rv:1.9.2.10) gecko/20100922 ubuntu/10.10 (Maverick) firefox/3.6.10 "," mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) applewebkit/537.36 (khtml, like Gecko) chrome/52.0.2743.116 safari/537.36 ","

The use of random proxies is as follows:

#-*-Coding:utf-8-*-
"Created on
April 20, 2017
This class is primarily used to generate random useragent
@author: Dzm
@see: https:// Github.com/qiyeboy/jiandan_2
@see: http://blog.csdn.net/tianjinjianzhan/article/details/51702232
@see: Http://outofmemory.cn/code-snippet/10668/scrapy--usage-random-user-agent-method
@keyword user-agent: User agent, Part of the HTTP protocol, which is part of the header domain, identifies the browser type and version, operating system and version, browser core, and other information accessed by the website
    through this identity, the site to which the user visits can display different typesetting
"'
import random From
eie.middlewares import user_agents

class Randomuseragent (object):

    def __init__ (self):
        Self.agents = user_agents. User_agents

    def process_request (self, request, spider):
        ua = Random.choice (self.agents)
        Request.headers.setdefault (' User-agent ', UA)