How to solve 403 error returned by Python crawler

This indicates that the website uses anti-crawling technology anti-web-crawlingtechnique (used by Amazon). it is relatively simple to check the user agent information.

Problem

When capturing data, the common debugging information is:

DEBUG: Crawled (200) 
 
   (referer: None)
 

If

DEBUG: Crawled (403) 
 
   (referer: None)
 

This indicates that the website uses anti-crawling technology anti-web-crawler technique (used by Amazon), which is easy to check the User Agent information.

Solution

Construct a User Agent in the request header as follows:

def start_requests(self):      yield Request("#",                    headers={'User-Agent': "your agent string"})

The above is the details about how the Python crawler returns the 403 error solution. For more information, see other related articles in the first PHP community!