Python Learning-crawler

Reprinted from a static blog

Most common download page

1 Import  2 response = Urllib2.urlopen ("http://www.baidu.com")3  print response.read ()

Post mode

1 ImportUrllib2 ImportUrllib23 4Values = {"username":"*****","Password":"*****"}5data =Urllib.urlencode (values)6URL ="   "7Request =urllib2. Request (url,data)8Response =Urlopen (request)9 PrintResponse.read ()

Get mode

1 ImportUrllib22 ImportUrllib3 4Values = {}5values["username"] =6values["Password"] =7data =UrlEncode (values)8URL =9Geturl = URL +"?"+DataTenRequest =urllib2. Request (geturl) oneResponse =Urllib2.urlopen (request) a PrintResponse.read ()

Set up Proxy

1 ImportUrllib22Enable_proxy =True3Proxy_handler = urllib2. Proxyhandler ({"http":'http://some-proxy.com:8080'})4Null_proxy_handler =urllib2. Proxyhandler ({})5 ifenable_proxy:6Opener =Urllib2.build_opener (proxy_handler)7 Else:8Opener =Urllib2.build_opener (null_proxy_handler)9Urllib2.install_opener (opener)

Set delay

1 Import Urllib2 2 response = Urllib2.urlopen ('http://www.baidu.com', data, 10)

Exception handling

1 ImportUrllib22 3req = urllib2. Request ('Http://blog.csdn.net/cqcre')4 Try:5 Urllib2.urlopen (req)6 excepturllib2. urlerror, E:7     ifHasattr (e,"Code"):8         PrintE.code9     ifHasattr (e,"reason"):Ten         PrintE.reason one Else: a     Print "OK"

Set cookies

1 ImportUrllib2 ImportUrllib23 ImportCookielib4 5filename ='Cookie.txt'6 #declares a Mozillacookiejar object instance to hold the cookie, and then writes the file7Cookie =cookielib. Mozillacookiejar (filename)8Opener =Urllib2.build_opener (urllib2. Httpcookieprocessor (cookie))9PostData =Urllib.urlencode ({Ten             'Stuid':'201200131012', one             'pwd':'23342321' a         }) - #URL of the login educational system -Loginurl ='Http://jwxt.sdu.edu.cn:7890/pls/wwwbks/bks_login2.login' the #impersonate the login and save the cookie to the variable -result =Opener.open (loginurl,postdata) - #Save cookies to Cookie.txt -Cookie.save (ignore_discard=true, ignore_expires=True) + #use cookies to request access to another url, which is the score search URL -Gradeurl ='Http://jwxt.sdu.edu.cn:7890/pls/wwwbks/bkscjcx.curscopre' + #request access to the results query URL aresult =Opener.open (gradeurl) at PrintResult.read ()

Python Learning-crawler