Crawling Web content with Python2 and Python3 masquerading browsers, respectively

Python Web crawling is very powerful, and using Urllib or URLLIB2 can easily crawl Web content. But most of the time we have to be aware that many sites are set up with anti-collection features, not so easy to grab the content you want.

Today I will share the download Python2 and python3 How to simulate the browser to skip the mask to crawl.

The most basic crawl:

#! /usr/bin/env python#-*-coding=utf-8-*-# @Author pythontabimport urllib.requesturl = "http://www.pythontab.com" html = u Rllib.request.urlopen (URL). Read () print (HTML)

But... Some sites can not be crawled, the anti-collection settings, so we have to change the method

Python2 (latest stable version python2.7)

#! /usr/bin/env python#-*-coding=utf-8-*-# @Author pythontab.comimport urllib2url= "http://pythontab.com" Req_header = {' User-agent ': ' mozilla/5.0 (Windows NT 6.1) applewebkit/537.11 (khtml, like Gecko) chrome/23.0.1271.64 safari/537.11 ', '             Accept ': ' text/html;q=0.9,*/*;q=0.8 ',             ' accept-charset ': ' iso-8859-1,utf-8;q=0.7,*;q=0.3 ',             ' Accept-encoding ': ' gzip ',             ' Connection ': ' Close ',             ' Referer ': None #注意如果依然不能抓取的话, here you can set the host to crawl the site             }req_ Timeout = 5req = Urllib2. Request (url,none,req_header) resp = Urllib2.urlopen (req,none,req_timeout) HTML = Resp.read () print (HTML)

Python3 (latest stable version python3.3)

#! /usr/bin/env python#-*-coding=utf-8-*-# @Author pythontabimport urllib.request  url = "Http://www.pythontab.com" headers = {' user-agent ': ' mozilla/5.0 (Windows NT 6.1) applewebkit/537.11 (khtml, like Gecko) chrome/23.0.1271.64 SAFARI/5 37.11 ',             ' Accept ': ' text/html;q=0.9,*/*;q=0.8 ',             ' accept-charset ': ' iso-8859-1,utf-8;q=0.7,*;q=0.3 ',             ' Accept-encoding ': ' gzip ',             ' Connection ': ' Close ',             ' Referer ': None #注意如果依然不能抓取, here you can set crawl site host             }  Opener = Urllib.request.build_opener () opener.addheaders = [headers]data = Opener.open (URL). Read () print (data)

