Details of the Python urllib module urlopen () and urlretrieve ()

1. urlopen () method
Urllib. urlopen (url [, data [, proxies]): Creates a class file object that represents a remote url, and then operates on this class file object like a local file to obtain remote data.
The url parameter indicates the path of the remote data, generally the url;
The data parameter indicates the data submitted to the url in post mode. (web users should know the two data submission methods: post and get. If you don't know, you don't have to worry too much about it. This parameter is rarely used in general );
The proxies parameter is used to set the proxy.
Urlopen returns a class object, which provides the following methods:
Read (), readline (), readlines (), fileno (), close (): these methods are used in the same way as file objects;
Info (): returns an httplib. HTTPMessage object, indicating the header information returned by the remote server.
Getcode (): return the Http status code. For an http request, 200 indicates that the request is successfully completed; 404 indicates that the URL is not found;
Geturl (): return the request url;
Sample Code:
Copy codeThe Code is as follows:
Import urllib

Url = "http://www.baidu.com /"

# Urlopen ()
Sock = urllib. urlopen (url)
HtmlCode = sock. read ()
Sock. close
Fp = open ("e:/1.html", "wb ")
Fp. write (htmlCode)
Fp. close

# Urlretrieve ()
Urllib. urlretrieve (url, 'e:/2.html ')

2. urlretrieve Method
Directly download the remote data to the local device.
Copy codeThe Code is as follows:
Urllib. urlretrieve (url [, filename [, reporthook [, data])
Parameter description:
Url: external or local url
Filename: Specifies the path to be saved locally (if this parameter is not specified, urllib will generate a temporary file to save data );
Reporthook: a callback function that triggers a callback when the server is connected and the corresponding data block has been transferred. We can use this callback function to display the current download progress.
Data: The data that is post to the server. This method returns a tuple (filename, headers) containing two elements. filename indicates the path saved to the local directory, and header indicates the response header of the server.

The following example demonstrates how to use this method. In this example, the html of the Sina homepage is crawled locally and saved in the D:/sina.html file, and the download progress is displayed.
Copy codeThe Code is as follows:
Import urllib
Def callbackfunc (blocknum, blocksize, totalsize ):
''' Callback function
@ Blocknum: The downloaded data block.
@ Blocksize: data block size
@ Totalsize: the size of the remote file.
'''
Percent = 100.0 * blocknum * blocksize/totalsize
If percent> 100:
Percent = 100.
Print "%. 2f %" % percent

Url = 'HTTP: // www.sina.com.cn'
Local = 'd: \ sina.html'
Urllib. urlretrieve (url, local, callbackfunc)