Python web module learning-urllib

Python web module learning-urllib
Prepare to write some columns of python web module learning, basically involves common web modules, including urllib, urllib2, httplib, urlparse, requests, now, let's start learning the first module. 1 urllib Introduction The python urllib module provides a method for getting webpage data from a specified URL address and then analyzing and processing it to obtain the desired data. 2 common method 2.1 urlopen -- create a class file object to read the specified URL help (urllib. urlopen) urlopen (url, data = None, proxies = None) Create a file-like object for the specified URL to read from. parameter: url: indicates the path of the remote data, which is generally the http or ftp path. Data: the data submitted to the url in get or post mode. Proxies: used for proxy settings. Python uses the urlopen function to obtain html data. urlopen returns a class object, which provides the following common methods: 1) read (), readline (), readlines (), fileno () and close (): these methods are used exactly the same as file objects. 2) info (): return an httplib. HTTPMessage object, indicating the header information returned by the remote server. 3) getcode (): return the Http status code. If it is an http request, 200 indicates that the request is successfully completed; 404 indicates that the URL is not found. 4) geturl (): return the url of the request. Code:

>>> Import urllib >>> response = urllib. urlopen ('HTTP: // www.51cto.com ') >>> res. read ()...... (A bunch of Web code) >>> res. readline () '<! DOCTYPE html PUBLIC "-// W3C // DTDXHTML 1.0 Transitional // EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> \ r \ n'... >>> Res. readlines ()... (List form of a bunch of Web code) >>> res.info () 
 
Urllib also provides some auxiliary methods for url encoding and decoding. There are no special symbols in the url, and some symbols have special purposes. We know that when we submit data in get mode, a string such as key = value will be added to the url, so '=' is not allowed in value ', therefore, it must be encoded. When the server receives these parameters, it must be decoded and restored to the original data. In this case, these auxiliary methods are useful: (mainly url encoding and decoding)-urllib. quote (string [, safe]): encode the string. The safe parameter specifies the character urllib that does not need to be encoded. unquote (string): decodes the string-urllib. quote_plus (string [, safe]): With urllib. quote is similar, but this method replaces ''with '+', while '% 20' is used to replace''-urllib. unquote_plus (string): decodes the string-urllib. urlencode (query [, doseq]): converts a list of dict or tuples containing two elements into url parameters. For example, the dictionary {'name': 'wklken', 'pwd': '000000'} will be converted to "name = wklken & pwd = 123" (commonly used) # Here we can combine it with urlopen to implement the post method and get method-urllib. pathname2url (path): Convert the local path to url path-urllib. url2pathname (path): url path conversion cost path with code:
>>> import urllib>>>res = urllib.quote('I am 51cto')>>> res'I%20am%2051cto'>>>urllib.unquote(res)'I am 51cto'>>>res = urllib.quote_plus('I am 51cto')>>> res'I+am+51cto'>>>urllib.unquote_plus(res)'I am 51cto'>>> params = {'name':'51cto','pwd':'51cto'}>>>urllib.urlencode(params)'pwd=51cto&name=51cto'>>>l2u=urllib.pathname2url('E:\51cto')'E%3A%29cto'>>>urllib.url2pathname(l2u)'E:)cto'
 
2.2 urlretrieve -- directly downloads remote data to the local help (urllib. urlretrieve) urlretrieve (url, filename = None, reporthook = None, data = None) parameter: url: Specifies the download URL finename: Specifies the local path to save (if the parameter is not specified, urllib generates a temporary file to save data. Reporthook: a callback function that triggers a callback when the server is connected and the corresponding data block is transferred. We can use this callback function to display the current download progress. Data: The post data to the server. This method returns a (filename, headers) tuples containing two elements. The following is an instance of the file downloaded by the urlretrieve method, which can display the download progress: #! /Usr/bin/env python #-*-coding: UTF-8-*-import urllibimport OS def schedule (a, B, c): ''' callback function @: downloaded data @ B: size of the data block @ c: size of the Remote File '''
Per = 100.0 * a * B/c if per> 100: per = 100 print "%. 2f % "% perurl = 'HTTP: // www.python.org/ftp/python/2.7.5/Python-2.7.5.tar.bz2'local = OS. path. join ('C: ', 'python-2.7.5.tar.bz2') urllib. urlretrieve (url, local, schedule) 2.3 urlcleanup -- clear because urllib. cache generated by urlretrieve ()
 
Through the above exercises, we can know that urlopen can easily obtain the remote html page information, and then analyze the required data through the python regular expression to match the desired data, then, use urlretrieve to download the data to the local device. You can use proxies (proxy method) to connect to a remote url address with limited access or connections. If the remote data volume is too large and the single-thread download is too slow, you can use multiple threads for download, this is the legendary crawler. The first two methods described above are the most common methods in urllib. These methods use the URLopener or FancyURLOpener class internally when obtaining remote data. As a user of urllib, we seldom use these two classes. If you are interested in urllib implementation or want urllib to support more protocols, you can study these two classes. In the Python manual, the author of urllib also lists the defects and shortcomings of this module. If you are interested, open the Python manual.