For details about how to use urllib in Python3 (header, proxy, timeout, authentication, exception handling), python3urllib

For details about how to use urllib in Python3 (header, proxy, timeout, authentication, exception handling), python3urllib

We can use urllib to capture remote data and save it. The following are several methods for python3 to capture webpage resources. For more information, see.

1. Simplest

import urllib.requestresponse = urllib.request.urlopen('http://python.org/')html = response.read()

2. Use Request

import urllib.requestreq = urllib.request.Request('http://python.org/')response = urllib.request.urlopen(req)the_page = response.read()

3. Send data

#! /usr/bin/env python3import urllib.parseimport urllib.requesturl = 'http://localhost/login.php'user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'values = {'act' : 'login','login[email]' : 'yzhang@i9i8.com','login[password]' : '123456'}data = urllib.parse.urlencode(values)req = urllib.request.Request(url, data)req.add_header('Referer', 'http://www.python.org/')response = urllib.request.urlopen(req)the_page = response.read()print(the_page.decode("utf8"))

4. Send data and headers

#! /usr/bin/env python3import urllib.parseimport urllib.requesturl = 'http://localhost/login.php'user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'values = {'act' : 'login','login[email]' : 'yzhang@i9i8.com','login[password]' : '123456'}headers = { 'User-Agent' : user_agent }data = urllib.parse.urlencode(values)req = urllib.request.Request(url, data, headers)response = urllib.request.urlopen(req)the_page = response.read()print(the_page.decode("utf8"))

5. http errors

#! /usr/bin/env python3import urllib.requestreq = urllib.request.Request('http://www.bkjia.com ')try:urllib.request.urlopen(req)except urllib.error.HTTPError as e:print(e.code)print(e.read().decode("utf8"))

6. Exception Handling 1

#! /usr/bin/env python3from urllib.request import Request, urlopenfrom urllib.error import URLError, HTTPErrorreq = Request("http://www.bkjia.com /")try:response = urlopen(req)except HTTPError as e:print('The server couldn't fulfill the request.')print('Error code: ', e.code)except URLError as e:print('We failed to reach a server.')print('Reason: ', e.reason)else:print("good!")print(response.read().decode("utf8"))

7. Exception Handling 2

#! /usr/bin/env python3from urllib.request import Request, urlopenfrom urllib.error import URLErrorreq = Request("http://www.bkjia.com /")try:response = urlopen(req)except URLError as e:if hasattr(e, 'reason'):print('We failed to reach a server.')print('Reason: ', e.reason)elif hasattr(e, 'code'):print('The server couldn't fulfill the request.')print('Error code: ', e.code)else:print("good!")print(response.read().decode("utf8"))

8. HTTP Authentication

#! /usr/bin/env python3import urllib.request# create a password managerpassword_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()# Add the username and password.# If we knew the realm, we could use it instead of None.top_level_url = "https://www.jb51.net /"password_mgr.add_password(None, top_level_url, 'rekfan', 'xxxxxx')handler = urllib.request.HTTPBasicAuthHandler(password_mgr)# create "opener" (OpenerDirector instance)opener = urllib.request.build_opener(handler)# use the opener to fetch a URLa_url = "https://www.jb51.net /"x = opener.open(a_url)print(x.read())# Install the opener.# Now all calls to urllib.request.urlopen use our opener.urllib.request.install_opener(opener)a = urllib.request.urlopen(a_url).read().decode('utf8')print(a)

9. Use a proxy

#! /usr/bin/env python3import urllib.requestproxy_support = urllib.request.ProxyHandler({'sock5': 'localhost:1080'})opener = urllib.request.build_opener(proxy_support)urllib.request.install_opener(opener)a = urllib.request.urlopen("http://www.bkjia.com ").read().decode("utf8")print(a)

10. Timeout

#! /usr/bin/env python3import socketimport urllib.request# timeout in secondstimeout = 2socket.setdefaulttimeout(timeout)# this call to urllib.request.urlopen now uses the default timeout# we have set in the socket modulereq = urllib.request.Request('http://www.bkjia.com /')a = urllib.request.urlopen(req).read()print(a)

Summary

The above is all the content of this article. I hope the content of this article will be helpful for everyone to learn or use python. If you have any questions, please leave a message.