Writing a web crawler in Python (iii): Handling of exceptions and classification of HTTP status codes

First of all, the HTTP exception handling problem.

When Urlopen cannot handle a response, a urlerror is generated.

However, the usual Python APIs such as Valueerror,typeerror can also be generated.

Httperror is a subclass of Urlerror that is typically generated in a specific HTTP URL.

1.URLError

Typically, urlerror occurs without a network connection (not routed to a specific server), or if the server does not exist.

In this case, the exception also has the "reason" attribute, which is a tuple (an array that can be understood as immutable),

Contains an error number and an error message.

Let's build a urllib2_test06.py to feel the exception handling:

Import urllib2  
      
req = urllib2. Request (' http://www.baibai.com ')  
      
try:urllib2.urlopen (req)  
      
except URLLIB2. Urlerror, E:    
    print E.reason

Press F5 to see what is printed:

[Errno 11001] getaddrinfo failed

In other words, the error number is 11001, the content is getaddrinfo failed

2.HTTPError

Each HTTP Reply object response on the server contains a number status code.

Sometimes the status code indicates that the server cannot complete the request. The default processor will handle part of this response for you.

For example, if response is a "redirect" that requires the client to obtain a document from another address, URLLIB2 will handle it for you.

Other urlopen that cannot be dealt with will produce a httperror.

Typical errors include "404" (pages cannot be found), "403" (Request Prohibition), and "401" (with authentication requests).

The HTTP status code represents the state of the response returned by the HTTP protocol.

For example, the client sends a request to the server, and if the requested resource is successfully obtained, the returned status code is 200, indicating a successful response.

If the requested resource does not exist, a 404 error is usually returned.