Python Learning Note (urllib module)

Python3 and Python2 Urllib module is not quite the same, this article is based on the premise of Python3.

Use of 1.urlopen

Import urllib.requesturllib.request.urlopen (URL, data=none, [Timeout,]*, Cafile=none, Capath=none, cadefault= False, context=None)#URL: the page that needs to be crawled #Data:post submitted data. Default is null, use a GET request, if data is a POST request #timeout: Set the site's access time-out

 import   Urllib.requestresponse  = Urllib.request.urlopen ( '  http://www.baidu.com   " )  print  (Response.read (). Decode ( " utf-8  Span style= "COLOR: #800000" > "  #  Span style= "COLOR: #008000" >response.read () gets the data format for the bytes type  #   requires decode (), converted to str type  

 #  post request   Import   urllib.parse  import   urllib.requestdata  = bytes (Urllib.parse.urlencode ({ "  word   ":  '  hello  } ', Encoding=

# Timeout Settings Import  = Urllib.request.urlopen ('http://httpbin.org/get', timeout=0.1)  Print(Response.read ())

Use of 2.Request

#GET RequestImporturllib.requestrequest= Urllib.request.Request ('https://python.org') Response=Urllib.request.urlopen (Request)Print(Response.read (). Decode ('Utf-8'))#POST Request fromUrllibImportrequest, parseURL='Http://httpbin.org/post'Headers= {    'user-agent':'mozilla/4.0 (compatible; MSIE 5.5; Windows NT)',    'Host':'httpbin.org'}dict= {    'name':'Germey'}data= Bytes (Parse.urlencode (dict), encoding='UTF8') Req= Request. Request (Url=url, Data=data, Headers=headers, method='POST') Response=Request.urlopen (req)Print(Response.read (). Decode ('Utf-8'))

3. Agent

ImportUrllib.requestproxy_handler=Urllib.request.ProxyHandler ({'http':'http://127.0.0.1:9743',    'HTTPS':'https://127.0.0.1:9743'}) Opener=Urllib.request.build_opener (proxy_handler) Response= Opener.open ('Http://httpbin.org/get')Print(Response.read (). Decode (' Utf-8 '))

4.Cookie

#Get CookiesImportHttp.cookiejar, Urllib.requestcookie=Http.cookiejar.CookieJar () handler=Urllib.request.HTTPCookieProcessor (Cookie) opener=Urllib.request.build_opener (handler) Response= Opener.open ('http://www.baidu.com') forIteminchCookies:Print(item.name+"="+item.value)#get a cookie and save it in a file#There are two types of formats, remember which format to save in which format to read good#format OneImportHttp.cookiejar, Urllib.requestfilename="Cookie.txt"Cookies=http.cookiejar.MozillaCookieJar (filename) handler=Urllib.request.HTTPCookieProcessor (Cookie) opener=Urllib.request.build_opener (handler) Response= Opener.open ('http://www.baidu.com') Cookie.save (Ignore_discard=true, ignore_expires=True)#format TwoImportHttp.cookiejar, Urllib.requestfilename='Cookie.txt'Cookies=http.cookiejar.LWPCookieJar (filename) handler=Urllib.request.HTTPCookieProcessor (Cookie) opener=Urllib.request.build_opener (handler) Response= Opener.open ('http://www.baidu.com') Cookie.save (Ignore_discard=true, ignore_expires=True)#second-reading cookies in format and access to URLsImportHttp.cookiejar, Urllib.requestcookie=Http.cookiejar.LWPCookieJar () cookie.load ('Cookie.txt', Ignore_discard=true, ignore_expires=True) Handler=Urllib.request.HTTPCookieProcessor (Cookie) opener=Urllib.request.build_opener (handler) Response= Opener.open ('http://www.baidu.com')Print(Response.read (). Decode ('Utf-8'))

5. Exception Handling

#Urllib.error has two error classes Urlerror and Httperror,httperror are subclasses of Urlerror, so it is common to catch small error classes and then catch large error classes fromUrllibImportRequest, ErrorTry: Response= Request.urlopen ('http://cuiqingcai.com/index.htm')excepterror. Httperror as E:Print(E.reason, E.code, E.headers, sep='\ n')excepterror. Urlerror as E:Print(E.reason)Else:    Print('Request successfully')

Python Learning Note (urllib module)