Python crawler (ii) Size and constraints of web crawler

Infi-chu:

http://www.cnblogs.com/Infi-chu/

First, the size of the Web crawler:

1. Small size, small amount of data, crawl speed is not sensitive, requests library, crawl Web page
2. Medium scale, large data size, crawl speed sensitive, scrapy library, crawl site
3. Large-scale, large-scale, search engine, crawl speed is critical, custom development, crawl the entire station

Second, robots agreement:

1. Meaning Robots Exclusion standard web crawler exclusions
2. Role: Web site tells the Web crawler which pages can be crawled, which do not
3. Form: Robots.txt file in the root directory of the website
4. Use:
A. Web crawler: Automatic or manual identification of robots.txt, and then content crawling
B. Binding: Can not follow, but pay attention to legal risks

Python crawler (ii) Size and constraints of web crawler