[Python] crawler Learning (1), python crawler Learning

[Python] crawler Learning (1), python crawler Learning

The basic operation of Web Crawlers is to capture webpages.

Start with the URL.

 

The URL is the http://www.baidu.com string entered on the browser side.

A URI consists of three parts:

① Resource Access naming mechanism;

② Host name for storing resources;

③ The Name Of The resource, represented by the path.

The URI is as follows:
Http://www.baidu.com/myhtml/html1223/

We can explain it as follows:

① This is a resource that can be accessed through the HTTP protocol,

② It is located on the host www.webmonkey.com.cn,

③ Access through the path "/html/html40.

A URL is a subset of a URI.

Generally speaking, URLs are strings used to describe information resources on the Internet and are mainly used in various WWW client programs and server programs.

URL can be used to describe various information resources in a unified format, including files, server addresses and directories.

The URL format is ([] with square brackets ):

Protocol: // hostname [: port]/path/[; parameters] [? Query] # fragment

 

The URL format consists of three parts:

① The first part is the protocol (or service mode ).

② The second part is the Host IP address (sometimes including the port number) that contains the resource ).

③ The third part is the specific address of the host resource, such as the Directory and file name.

The first part and the second part are separated by the ": //" symbol,

The second part and the third part are separated by the "/" symbol.

The first part and the second part are indispensable, and the third part can be omitted sometimes.

 

5. simple comparison between URL and URI

URI is a lower-level URL abstraction, a string text standard.

In other words, URI belongs to the parent class, and URL belongs to the URI subclass. A URL is a subset of a URI.

URI is defined as a uniform resource identifier;

The URL is defined as a uniform resource identifier.

The difference between the two is that URI represents the path of the Request server and defines such a resource.

The URL also describes how to access this resource (http ://).

 

The following is a small example of two URLs.

 

1. Example of http url:
The super text transmission protocol HTTP is used to provide resources for the Super Text Information Service.

Example: http://www.peopledaily.com.cn/channel/welcome.htm

Its computer domain name is www.peopledaily.com.cn.

The super Startup File (.html) is welcome.htm under the directory/channel.

This is a computer of the People's Daily of China.

Example: http://www.rol.cn.NET/talk/talk1.htm

The computer domain name is www.rol.cn. Net.

The super Startup File (.html) is the talk1.htm under the directory/talk.

This is the address of the reide chat room. You can enter room 1st of the reide chat room.

2. File URL
When a file is expressed as a URL, the server uses file to indicate the Host IP address, file access path (directory), and file name.

Sometimes the Directory and file name can be omitted, but the "/" symbol cannot be omitted.

Example: file: // ftp.yoyodyne.com/pub/files/foobar.txt

The above URL indicates a file in the pub/files/directory of the ftp.yoyodyne.com host. The file name is foobar.txt.

Example: file: // ftp.yoyodyne.com/pub

Indicates the directory/pub on the ftp.yodyne.com host.

Example: file: // ftp.yoyodyne.com/

Represents the root directory of the ftp.yoyodyne.com host.

 

The most important processing object of a crawler is the URL. It obtains the required file content based on the URL and further processes it.

Therefore, an accurate understanding of URLs is critical to understanding web crawlers.

Original: http://blog.csdn.net/pleasecallmewhy/article/details/8922826