Web crawling Based on python crawling practices

Python comes with the urllib, urllib2 modules, and third-party requests library to capture webpages. Here we use the easy_install package management tool to download the requests library and BeautifulSoup library. Under the CMD command line, switch to the easy_install directory and run the command easy_install package name.

Easy_install requests

After the requests package is installed, we can use urllib, urllib2, or the requests library to capture the web page.

1. webpage content capturing

#! /usr/bin/env python#coding:utf-8import urllibimport urllib2import requestsimport sysurl = 'http://www.csdn.net'def urllib2Test():req = urllib2.Request(url)response = urllib2.urlopen(req)thePage = response.read()def requestsTest():r = requests.get(url)r.status_coder.contentr.headersdef urllib2TestEx(url):req = urllib2.Request(url)try:response = urllib2.urlopen(req)content = response.read()except urllib2.URLError,e:print e.reasondef urlhttperror(url):req = urllib2.Request(url)try:urllib2.urlopen(req)except urllib2.HTTPError,e:print e.read()if __name__ == '__main__':urllib2Test()requestsTest()urllib2TestEx(url)urlhttperror(url)

2. crawler disguised as browser access

For websites that prohibit crawlers, You can disguise them as browser access and add UserAgent information to the request. Add and modify headers = {'user-agent': 'mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv: 1.9.1.6) gecko/20091201 Firefox/3.5.6 '} req = urllib2.Request ("http://blog.csdn.net/nevasun", headers = headers)
Disguised as a browser for access as follows

#! /Usr/bin/env python # coding: utf-8import requestsfrom BeautifulSoup import BeautifulSoupfrom OS. path import dirname, abspathimport sysimport OS # PREFIX = dirname (abspath (_ file _) # This code is used to solve the problem of Chinese error reload (sys) sys. setdefaultencoding ("utf8 ") ######################################## ############ defaultWaitTime = 1def getHtmlContent (url): global defaultWaitTimecontent = Noneretry = 0 headers = {'user-agent': 'mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) chrome/31.0.1650.57 Safari/537.36 '} AccessFrequency = defaultWaitTimewhile retry <5: try: r = requests. get (url, timeout = 10, headers = headers) content = r. contentreturn contentexcept: retry + = 1time. sleep (AccessFrequency) return contentdef Test (): content = getHtmlContent ("http://blog.csdn.net/figo829/article/details/18015537") # print contentif _ name _ = '_ main _': Test ()