Urllib Module Use simple example

First, the simplest use

Import= Urllib2.urlopen ("https://www.baidu.com")Print Response.read ()

View Code

Second, construct the request object

Request = Urllib2. Request ("https://www.baidu.com"= urllib2.urlopen (request)Print Response.read ()

View Code

Third, through the post, GET way request

POST

Values = {'username':'Test','Passwrod':'123'}data=Urllib.urlencode (values)PrintData#username=test&passwrod=123Request = Urllib2. Request ("https://www.baidu.com", Data=data) Response=Urllib2.urlopen (Request)PrintResponse.read ()

View Code

GET

Value ={}value['username']='Test'value['Password']='123'Data=urllib.urlencode (value) URL="https://www.baidu.com"+"?"+DataPrintUrl#https://www.baidu.com?username=test&password=123Request = Urllib2. Request (url=URL) Response=Urllib2.urlopen (Request)PrintResponse.read ()

View Code

Iv. Quote, encode

' haha '  = urllib.quote (a)print= urllib.unquote (a)print B

View Code

UrlEncode The get part of the three has a sample

V. Setting the header of the request headers

URL ="https://www.baidu.com"value= {"username":"Test","Password":"123"}data=Urllib.urlencode (value) header= {        "user-agent":"mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) gecko/20100101 firefox/50.0",        "Referer":"http://tieba.baidu.com/f?kw=%E4%BF%9D%E5%AE%9A&ie=utf-8&pn=50"}request= Urllib2. Request (url=url,data=data,headers=header) Response=Urllib2.urlopen (Request)PrintResponse.read ()

View Code

Urlopen is urllib2. An instance of Openerdirector, a opener, a special default opener. Therefore, this opener does not always meet our needs,
At this time, we need to construct our own opener.

SOURCE Excerpt

_opener = Nonedef  Install_opener (opener):    Global  _opener    = opener # ———————————————————————————————————————————————— def Urllopen ():     """ ... """    return opener.open (URL, data, timeout)

View Code

Vi. setting up an agent

Enable_proxy =Trueproxy_handler= Urllib2. Proxyhandler ({"http":'http://some-proxy.com:8080'}) Null_proxy=Urllib2. Proxyhandler ({})ifEnable_proxy:opener= Urllib2.build_opener (Proxy_handler)#Create a Opener objectElse: Opener=Urllib2.build_opener (null_proxy)#Urllib2.install_opener (opener)#apply the opener globallyRequest= Urllib2. Request ("https://www.baidu.com") Response=Opener.open (Request) Response=Urllib2.urlopen (Request)# PrintResponse.read ()

View Code

Vii. operation of Cookies

ImportCookielib#Create a Cookiejar instance to save the cookieCookie =Cookielib. Cookiejar ()#Creating a Cookie processorHandler =Urllib2. Httpcookieprocessor (Cookie)#Create a openerOpener =Urllib2.build_opener (handler)#request a URL with a opener with a cookie processorResponse = Opener.open ("https://www.baidu.com")#  forIteminchCookies:PrintItem#<cookie bidupsid=25441729620bf793c1be08ca0b43c8d4 for .baidu.com/>    Print 'Name ='+item.name#Name = Bidupsid    Print 'Value ='+item.value#Value = 25441729620bf793c1be08ca0b43c8d4

View Code

Viii. saving cookies to a file

ImportCookielibfilename="/home/an/savecookie.test"#Create a Mozillacookiejar object to hold the cookie and later write to the objectCookie =Cookielib. Mozillacookiejar (filename)#Creating a cookie processorHandle =Urllib2. Httpcookieprocessor (Cookie)#Build HandlerOpener =Urllib2.build_opener (handle) Response= Opener.open ("http://www.baidu.com")#Save cookies to fileCookie.save (ignore_discard=true,ignore_expires=True)#Ignore_discard is saved even if the cookie is discarded. #Ignore_expires If the cookie in the file already exists, then overwrite

View Code

Nine, remove the cookie from the file and use

Import= cookielib. Mozillacookiejar () cookie.load ("/home/an/savecookie.test", Ignore_expires=true, ignore_discard=== = Urllib2. Request ("http://www.baidu.com"= opener.open (request)  Print Response.read ()

View Code

Urllib Module Use simple example