General methods for adding headers to the 01-urllib library

2018-08-23 13:07:57

For some Web sites, we need to add a request header to complete the crawl of the page, otherwise we will get some errors, unable to return to crawl Web pages. Here are two ways to add a request header.

Method One: Complete with Build_opener and addheaders

1 Importurllib.request2Url="http://www.meizitu.com"3 #Note: Headers is a tuple in Urllib4Headers= ("user-agent","mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/63.0.3239.132 safari/537.36 QIHU 360SE")5 6Opener=Urllib.request.build_opener ()7opener.addheaders=[Headers]8Data=opener.open (URL)9 Print(Data.read ())

Note: The headers here is to be written as a tuple type. If you write as a dictionary type, you will get an error!

Method Two, create a request instance object

1 #Case 12 Importurllib.request3Url="http://www.meizitu.com"4 #Note: In Urllib this kind of headers is needed to be a dictionary5headers={"user-agent":"mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/63.0.3239.132 safari/537.36 QIHU 360SE"}6Req=urllib.request.request (url=url,headers=headers)7file=Urllib.request.urlopen (req)8 9 #If there are some decoding errors, add "ignore" to it .Ten Print(File.read (). Decode ("Utf-8",'Ignore'))

• Note: The headers here should be written as a dictionary type.
• Create a Reques object, put the required headers,url,proxy in, or in the post request can also put the encoded data value in, and then open with Urlopen, it is more convenient.

In addition, this method can also use Add_headers () to add headers, the code is as follows:

1 Importurllib.request2 Try:3Url="http://www.meizitu.com"4     5Req=urllib.request.request (url=URL)6  7Req.add_header ("user-agent","mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/63.0.3239.132 safari/537.36 QIHU 360SE")8File=urllib.request.urlopen (req,timeout=10.1)9 Ten     Print(File.read (). Decode ("Utf-8",'Ignore')) One exceptException as E: A     Print("Time-out", str (e))

Summary: Through the above two methods, you can complete the parameters of the request header settings, but note that headers is the dictionary type to pass in or the tuple type.

General methods for adding headers to the 01-urllib library