Definition and URL composition of a python crawler without basic writing

As the saying goes, to do a good job, you must first sharpen the tool. as the first article in this series, we also need to sharpen the tool to introduce the definition of crawlers and the basic knowledge needed to write crawlers. 1. definition of web crawler

Web crawlers, that is, Web Spider, are an image name.
Comparing the Internet to a Spider, a Spider is a web crawler.
Web crawlers search for web pages based on their link addresses.
Read the content of a webpage from a page (usually the homepage) of a website and find other links on the webpage,
Search for the next webpage through these links until all the webpages of the website are crawled.
If the whole internet is regarded as a website, the web spider can use this principle to capture all the web pages on the Internet.
In this case, Web crawler is a crawling program and a crawling program.
The basic operation of Web crawlers is to capture webpages.
So how can we get the page we want as we like?
Start with the URL.

II. webpage browsing process

In fact, the process of capturing a webpage is the same as that of browsing a webpage through the IE browser.
For example, enter www.baidu.com in the address bar of your browser.
The process of opening a webpage is actually that the browser acts as a browser "client" and sends a request to the server, "grabbing" the server file, and then explaining and presenting it.
HTML is a markup language that uses tags to tag content and parse and distinguish it.
The function of the browser is to parse the obtained HTML code, and then convert the original code into a website page that we can directly see.

3. concepts and examples of URI and URL

Simply put, the URL is the http://www.baidu.com string entered on the browser side.
Before you understand a URL, you must first understand the concept of URI.
What is URI?
Each type of available resources on the Web, such as HTML documents, images, video clips, and programs, are located by a Universal Resource Identifier (URI.
A uri consists of three parts:
① Resource access naming mechanism;
② Host name for storing resources;
③ The name of the resource, represented by the path.
The URI is as follows:
Http://www.why.com.cn/myhtml/html1223/
We can explain it as follows:
① This is a resource that can be accessed through the HTTP protocol,
② It is located on the host www.webmonkey.com.cn,
③ Access through the path "/html/html40.

IV. URL comprehension and examples

A url is a subset of a URI. It is the abbreviation of Uniform Resource Locator ".
Generally speaking, URLs are strings used to describe information resources on the Internet and are mainly used in various WWW client programs and server programs.
URL can be used to describe various information resources in a unified format, including files, server addresses and directories.
The URL format is ([] with square brackets ):
Protocol: // hostname [: port]/path/[; parameters] [? Query] # fragment

The URL format consists of three parts:
① The first part is the protocol (or service mode ).
② The second part is the host IP address (sometimes including the port number) that contains the resource ).
③ The third part is the specific address of the host resource, such as the Directory and file name.
The first part and the second part are separated by the ": //" symbol,
The second part and the third part are separated by the "/" symbol.
The first part and the second part are indispensable, and the third part can be omitted sometimes.

5. simple comparison between URL and URI

URI is a lower-level URL abstraction, a string text standard.
In other words, URI belongs to the parent class, and URL belongs to the URI subclass. A url is a subset of a URI.
URI is defined as a uniform resource identifier;
The URL is defined as a uniform resource identifier.
The difference between the two is that URI represents the path of the request server and defines such a resource.
The URL also describes how to access this resource (http ://).

The following is a small example of two URLs.

1. example of http url:
The Super text transmission protocol HTTP is used to provide resources for the super text information service.
Example: http://www.peopledaily.com.cn/channel/welcome.htm
Its computer domain name is www.peopledaily.com.cn.
The Super startup file (.html) is welcome.htm under the Directory/channel.
This is a computer of the People's Daily of China.
Example: http://www.rol.cn.net/talk/talk1.htm
The computer domain name is www.rol.cn.net.
The Super startup file (.html) is the talk1.htm under the Directory/talk.
This is the address of the Reide chat room. you can enter room 1st of the Reide chat room.

2. file URL
When a file is expressed as a URL, the server uses file to indicate the host IP address, file access path (directory), and file name.
Sometimes the directory and file name can be omitted, but the "/" symbol cannot be omitted.
Example: file: // ftp.yoyodyne.com/pub/files/foobar.txt
The above URL indicates a file in the pub/files/directory of the ftp.yoyodyne.com host. the file name is foobar.txt.
Example: file: // ftp.yoyodyne.com/pub
Indicates the directory/pub on the ftp.yodyne.com host.
Example: file: // ftp.yoyodyne.com/
Represents the root directory of the ftp.yoyodyne.com host.

The most important processing object of a crawler is the URL. it obtains the required file content based on the URL and further processes it.
Therefore, an accurate understanding of URLs is essential to understanding web crawlers.

Now, we will introduce the basic knowledge here. next we will perform some actual crawler operations.