Python3 Web crawler Notes

Resources

• Official Python documentation
• Related information (1) This very good, easy-to-read overview of the entire Python learning framework.
• Related Information (2)

Code implementation (i): Crawl the specified page with Python

1 #!/usr/bin/env python2 #Encoding:utf-83 Importurllib.request4  5URL ="http://www.baidu.com"6data =urllib.request.urlopen (URL). Read ()7data = Data.decode ('UTF-8')8 Print(data)

Urllib.request is a library that belongs to Urllib. Click here to open the official related documentation. How should official documents be used? First point just mentioned this link into the page has urllib of several sub-libraries, we temporarily use the request, so we first look at the Urllib.request section. The first thing you see is a word about what this library is for:

The Urllib.request module defines functions and classes which to opening URLs (mostly HTTP) in a complex world-basi C and Digest authentication, redirections, cookies and more.

Then we read the Urlopen () function part of our code.

Urllib.request.urlopen (URL, data=none, [Timeout,]*, Cafile=none, Capath=none, Cadefault=false)

The key part is the return value, which returns a Http.client.HTTPResponse object that has various methods, such as the Read () method that we use, which can be chained to the official document link. According to the official documentation, I run the above program with the console and continue to run the following code to be more familiar with what these messy methods do.

 >>> import   urllib.request  >>> a = Urllib.request.urlopen ( " http://10.54.0.2/oaapp/webobjects/oaapp.woa    )  >>> type (a)  <class   " http.client.httpresponse   " >>>> A.geturl ()  Span style= "color: #800000;" > '  

Python3 Web crawler Notes