Ip address banned during crawling This post was last edited by zzfloud from 2013-11-2422: 36: 33. Recently, I used snoopy to capture the content of a website, after I first found that the ip address was blocked, I changed the user-agent to google's spider according to the solution on the Internet.
At the end of this post, zzfkyo edited the content of a website to be crawled recently at 22:36:33 on November 24,. I used snoopy to capture the content of the website. After I first found that an ip address was blocked, I switched the user-agent to google's spider according to the solution on the Internet, and forged an ip address with snoopy (a random ip address is changed for every capture) however, after capturing more than one hundred pages, it is still blocked by the ip address, and thus cannot be crawled. Is there any good solution? Snoopy capture an ip address and share it: More
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.