How to disguise and escape anti-crawler programs in python web crawler

Source: Internet
Author: User
Tags python web crawler

How to disguise and escape anti-crawler programs in python web crawler

Sometimes, the crawler code we have written is still running well, And suddenly an error is reported.

The error message is as follows:

Http 800 Internal internet error

This is because your object website has configured anti-crawler programs. If you use the existing crawler code, it will be rejected.

The previous normal crawler code is as follows:

From urllib. request import urlopen..html = urlopen (scrapeUrl) bsObj = BeautifulSoup (html. read (), "html. parser ")

At this time, we need to disguise our crawler code,

Add a header to it to disguise it as a request from a browser.

The modified code is as follows:

Import urllib. parseimport urllib. requestfrom bs4 import BeautifulSoup... req = urllib. request. request (scrapeUrl) req. add_header ('user-agent', 'mozilla/4.0 (compatible; MSIE 5.5; Windows NT) ') response = urllib. request. urlopen (req) html = response. read () bsObj = BeautifulSoup (html, "html. parser ")

OK, everything is done, and you can continue crawling.

The above is all the content of this article. I hope it will be helpful for your learning and support for helping customers.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.