Python web crawler (i): the definition of web crawler

The web crawler, the spider, is a very vivid name.

The internet is likened to a spider's web, so spiders are crawling around the web.
Web spiders are looking for Web pages through the URL of a Web page.

From one page of the site (usually the homepage), read the contents of the Web page, find the other link address in the page,

Then look for the next page through these link addresses, and keep looping until all pages of the site have been crawled.

If the entire Internet as a Web site, then the network spider can use this principle to the Internet all the pages are crawled down.

In this way, the web crawler is a crawling program, a crawl Web page program.

The basic operation of web crawler is to crawl Web pages.

Python web crawler (i): the definition of web crawler