How python crawlers work: python Crawlers

How python crawlers work: python Crawlers

1. How crawlers work

Web crawlers, that is, Web Spider, are an image name. Comparing the Internet to a Spider, a Spider is a web crawler. Web crawlers search for Web pages based on their link addresses. Starting from a website page (usually the homepage), read the content of the webpage, find other link addresses in the webpage, and then find the next Webpage through these link addresses. This keeps repeating, until all the web pages of the website are crawled. If the whole Internet is regarded as a website, the web spider can use this principle to capture all the web pages on the Internet. In this case, web crawler is a crawling program and a crawling program. The basic operation of Web Crawlers is to capture webpages. So how can we get the page we want as we like? Start with the URL.

In fact, the process of capturing a webpage is the same as that of browsing a webpage through the IE browser. For example, enter www.baidu.com in the address bar of your browser. The process of opening a webpage is actually that the browser acts as a browser "client" and sends a request to the server, "grabbing" the server file, and then explaining and presenting it. HTML is a markup language that uses tags to tag content and parse and distinguish it. The function of the browser is to parse the obtained HTML code, and then convert the original code into a website page that we can directly see.

Simply put, the URL is the http://www.baidu.com string entered on the browser side. Before you understand a URL, you must first understand the concept of URI.

What is URI?

Each type of available resources on the Web, such as HTML documents, images, video clips, and programs, are located by a Universal Resource Identifier (URI.

A URI consists of three parts:

• Resource Access naming mechanism;
• Host Name for storing resources;
• The name of the resource, represented by the path.

The following URI: http://www.why.com.cn/myhtml/html1223/

• This is a resource that can be accessed through the HTTP protocol,
• On the host www.webmonkey.com.cn,
• Access through the path "/html/html40.

2. URL comprehension and Examples

A URL is a subset of a URI. It is the abbreviation of Uniform Resource Locator ". Generally speaking, URLs are strings used to describe information resources on the Internet and are mainly used in various WWW client programs and server programs. URL can be used to describe various information resources in a unified format, including files, server addresses and directories. The URL format is ([] with square brackets ):

Protocol: // hostname [: port]/path/[; parameters] [? Query] # fragment

The URL format consists of three parts:

• The first part is the protocol (or service mode ).
• The second part is the Host IP address (sometimes including the port number) that contains the resource ).
• The third part is the specific address of the host resource, such as the Directory and file name.

The first part and the second part are separated by the ": //" symbol, and the second and third parts are separated by the "/" symbol. The first part and the second part are indispensable, and the third part can be omitted sometimes.

3. simple comparison of URL and URI

URI is a lower-level URL abstraction, a string text standard. In other words, URI belongs to the parent class, and URL belongs to the URI subclass. A URL is a subset of a URI. The URI is defined as a uniform resource identifier, and the URL is defined as a uniform resource identifier. The difference between the two is that URI represents the path of the Request server and defines such a resource. The URL also describes how to access this resource (http ://).

The following is a small example of two URLs.

1. Example of http url:

The super text transmission protocol HTTP is used to provide resources for the Super Text Information Service.

Example: http://www.peopledaily.com.cn/channel/welcome.htm

Its computer domain name is www.peopledaily.com.cn.

The super Startup File (.html) is welcome.htm under the directory/channel.

This is a computer of the People's Daily of China.

Example: http://www.rol.cn.NET/talk/talk1.htm

The computer domain name is www.rol.cn. Net.

The super Startup File (.html) is the talk1.htm under the directory/talk.

This is the address of the reide chat room. You can enter room 1st of the reide chat room.

2. File URL

When a file is expressed as a URL, the server uses file to indicate the Host IP address, file access path (directory), and file name.

Sometimes the Directory and file name can be omitted, but the "/" symbol cannot be omitted.

Example: file: // ftp.yoyodyne.com/pub/files/foobar.txt

The above URL indicates a file in the pub/files/directory of the ftp.yoyodyne.com host. The file name is foobar.txt.

Example: file: // ftp.yoyodyne.com/pub

Indicates the directory/pub on the ftp.yodyne.com host.

Example: file: // ftp.yoyodyne.com/

Represents the root directory of the ftp.yoyodyne.com host.

The most important processing object of a crawler is the URL. It obtains the required file content based on the URL and further processes it.

Therefore, an accurate understanding of URLs is essential to understanding web crawlers.

The above is all the content of this article. I hope this article will help you in your study or work. I also hope to provide more support to the customer's home!