Write a web crawler in Python--0 basics

Here are a few things to do before crawling a Web site

1. Download and check the Web site's robots.txt file to let the crawler know what restrictions the site crawls.

2. Check site Map

3. Estimating Site Size

use Baidu or Google search Site:example.webscraping.com

The results are as follows

Find related results in about 5

The number is the estimated value. Site administrators need to know more about the exact amount of index

4. identify the technology used by the site

using the Builtwith module in Python

Https://pypi.python.org/pypi/python-builtwith

Run pip install Builtuith

after installation is complete using Commands

I mport Builtwith

B uiltwith.parse ('http://example.webscraping.com')

Write a web crawler in Python--0 basics