The code of Ethics for Python Crawlers---robots protocol

Before writing a crawler to crawl data, in order to avoid some of the copyright data later brought about a lot of legal issues,

You can avoid crawling certain pages by viewing the robots.txt file for your Web site.

Robots protocol, inform the crawler and other search engines those pages can crawl, which can not. It's just a passing moral code,

There is no mandatory provision, which is fully complied with by individual will. As a moral technician, abide by the robots agreement,

Help build a better internet environment.

The robots file address of the website is usually robots.txt after the homepage is added, such as Www.taobao.com/robots.txt

A simple to determine whether the user agent complies with the robots file requirements of the small program, eligible to download the webpage:

Import robotparserimport urllib2def download (url, user_agent= ' wswp ',  num_retries=2):     print  ' downloading: ',  url    headers = {' User-agent ':  user_agent}    request = urllib2. Request (url, headers=headers)     try:         Html = urllib2.urlopen (Request). Read ()     except urllib2. urlerror as e:        print  ' Download error: ',  e.reason        html = None         if num_retries > 0:            if hasattr (e,  ' code ')  and 500 <= e.code < 600:                 return download (url,num_retries-1)     return htmldef can _be_download (url, user_agent= ' wswp)               #设置一个默认的用户代理     rp = robotparser. Robotfileparser ()     url = url.split ('/') [2]                  #获取主页网址     rp.set_url ('/HTTP '  + str (URL)  +  '/robots.txt ')    #robots. txt address     rp.read ()     if rp.can_fetch (user_agent= ' wswp ',  url):         download (URL)

The code of Ethics for Python Crawlers---robots protocol