Python crawler (1): basic concepts and basic concepts of python Crawler

Python crawler (1): basic concepts and basic concepts of python Crawler
Web crawlers are defined as Web Crawlers (Web Spider, also known as Web Spider, Web robot, and Web page chaser). Web crawlers follow certain rules, programs or scripts that automatically capture World Wide Web information. In addition, some frequently used names include ant, automatic indexing, simulation programs, and worms. If you think of the Internet as a Spider, Spider is a web crawler. Web crawlers search for Web pages based on their link addresses. Starting from a website page (usually the homepage), read the content of the webpage, find other link addresses in the webpage, and then find the next Webpage through these link addresses. This keeps repeating, until all the web pages of the website are crawled. If the whole Internet is regarded as a website, the web spider can use this principle to capture all the web pages on the Internet. In this case, web crawler is a crawling program and a crawling program.
Simply put, the basic task of Web Crawlers is to capture web content..
In fact, the process of capturing a webpage using the URL concept is the same as that of browsing a webpage using the IE browser. For example, enter www.baidu.com in the address bar of your browser. The process of opening a webpage is actually that the browser acts as a browser "client" and sends a request to the server, "grabbing" the server file, and then explaining and presenting it. HTML is a markup language that uses tags to tag content and parse and distinguish it. The function of the browser is to parse the obtained HTML code, and then convert the original code into a website page that we can directly see.
Before understanding the URL, you must first understandURI Concept.
Each type of available resources on the Web, such as HTML documents, images, video clips, and programs, are located by a Universal Resource Identifier (URI. A URI consists of three parts:
① Resource Access naming mechanism;
② Host name for storing resources;
③ The Name Of The resource, represented by the path.
For example, URI: http://www.baidu.com.cn/myhtml/html1223/, we can explain the following:
① This is a resource that can be accessed through the HTTP protocol,
② On the host www.baidu.com.cn,
③ Access through the path "/html/html40.
URL ConceptA URL is a subset of a URI.. It is the abbreviation of Uniform Resource Locator ". Generally speaking, URLs are strings used to describe information resources on the Internet and are mainly used in various WWW client programs and server programs. URL can be used to describe various information resources in a unified format, including files, server addresses and directories.

URL example

1. Example of http url:
The super text transmission protocol HTTP is used to provide resources for the Super Text Information Service.

Example: http://www.peopledaily.com.cn/channel/welcome.htm
Its computer domain name is www.peopledaily.com.cn.
The super Startup File (.html) is welcome.htm under the directory/channel.
This is a computer of the People's Daily of China.

Example: http://www.rol.cn.net/talk/talk1.htm
The computer domain name is www.rol.cn.net.
The super Startup File (.html) is the talk1.htm under the directory/talk.
This is the address of the reide chat room. You can enter room 1st of the reide chat room.


2. File URL
When a file is expressed as a URL, the server uses file to indicate the Host IP address, file access path (directory), and file name.
Sometimes the Directory and file name can be omitted, but the "/" symbol cannot be omitted.
Example: file: // ftp.yoyodyne.com/pub/files/foobar.txt
The above URL indicates a file in the pub/files/directory of the ftp.yoyodyne.com host. The file name is foobar.txt.

Example: file: // ftp.yoyodyne.com/pub
Indicates the directory/pub on the ftp.yodyne.com host.

Example: file: // ftp.yoyodyne.com/
Represents the root directory of the ftp.yoyodyne.com host.