Python3 using Urllib module to make web crawler

Urllib

Urllib module is a URL processing package for Python3

which

1, urllib.request mainly open and read the URLs

The main use of personal 1:

Open the corresponding URL:urllib.request.open (URL)

Use Urllib.request.build_opener ([handler, ...]) to disguise as a corresponding browser

Import urllib# to disguise as a browser (I'm using chrome) headers = (' User-agent ', ' mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/46.0.2490.86 safari/537.36 ') url= ' http://hotels.ctrip.com/' Opener = Urllib.request.build_opener () #将要伪装成的浏览器添加到对应的http头部opener. addheaders=[headers] #读取相应的urldata = Opener.open (URL). Read () #将获得的html解码为utf -8data=data.decode (' Utf-8 ') print (data)

2, Urllib.parse is mainly used to resolve the URL

Main methods:

Urllib.parse.urlparse (urlstring)

function: resolves the corresponding URL into six parts and returns it in the tuple's data format. (almost identical in function to Urlsplit ())

Import Urllibo = Urllib.parse.urlparse (' http://www.cwi.nl:80/%7Eguido/Python.html ') print (o) print (o.path) print ( O.scheme) print (o.port) print (O.geturl ())

The corresponding result:

Parseresult (scheme= ' http ', netloc= ' www.cwi.nl:80 ', path= '/%7eguido/python.html ', params= ', query= ', fragment= ')
/%7eguido/python.html
http
80
Http://www.cwi.nl:80/%7Eguido/Python.html

2. Build a new Url--urllib.parse.urljoin (base, URL)

Parameters: Base: Basic URL link

URL: Another URL

From Urllib.parse import urljoina=urljoin (' http://www.cwi.nl/%7Eguido/Python.html ', ' faq.html ') print (a)

Results: http://www.cwi.nl/%7Eguido/FAQ.html

This function in the crawler should be more convenient, I used to be a rather stupid method of direct string concatenation

3. Exception Handling Urllib.error

Use try-except to catch exceptions

The main wrong way is two kinds of urlerror and Httperror

Because Httperror is Urlerror subclass, so urlerror should be written in the httperror behind, plainly is to find the son must know father, find father, not necessarily know son.

Try:  data=urllib.request.urlopen (URL)  print (Data.read (). Decode (' Utf-8 ')) except Urllib.error.HTTPError as E:  print (E.code) except Urllib.error.URLError as E:  print (E.reason)

Result: [Winerror 10060] The connection attempt failed because the connecting party did not respond correctly after a period of time or the connected host was unresponsive.

If Httperror is captured, the code is output and the Urlerror exception is not processed. If it is not httperror, it will catch the urlerror exception, the output error cause

