Crawler Basics---HTTP protocol understanding, Web-based basics, crawler fundamentals

I. Understanding URLs and URIs for HTTP protocols

Before learning HTTP, we need to know the URL, URI (exact description of the location of a resource and if you want to access it)

Url:universal Resource Locator Uniform Resource Locator that describes a specific location of a resource on a particular server.

Uri:uniform Resource Identifier Uniform Resource Identifier, URI has two forms of URL and urn.

HTTP and HTTPS

We often see HTTP or HTTPS at the header of the URL, which is the type of protocol required to access the resource, in addition to HTTP and HTTPS and FTP,SFTP,SMB. In reptiles, we use HTTP and HTTPS.

Http:hypertext Transfer Protocol Hypertext Transfer Protocol is one of the most widely used network protocols on the Internet. All WWW documents are subject to this standard. HTTP is a client and server-side request and response standard, it uses Socke based on TCP, so HTTP is a reliable data transfer protocol, because it is a short link, that is, a request, a response is broken, so it shows a stateless phenomenon. The current version of HTTP 1.1 is widely used.

The Https:hyper Text Transfer Protocol over secure Socket layer is a security-targeted HTTP channel, which is simply the secure version of HTTP, which is the SSL layer under HTTP, referred to as HTTPS. The security base for HTTPS is SSL, so the content he transmits is SSL-encrypted, and its main role is:

1. Establish an information security channel to ensure the security of data transmission
2. Confirm the authenticity of the website, all use of HTTPS site, you can click on the browser address bar lock logo to see the real information after the site certification, but also through the CA authority issued by the security seal to inquire.

The difference between HTTP and https:

1. The HTTPS protocol requires a certificate to be applied to the CA, and the general free certificate is very small and requires a fee.
2. HTTP is a Hypertext Transfer Protocol, the information is plaintext transmission, HTTPS is a secure SSL encryption transport protocol.
3. HTTP and HTTPS use a completely different connection, the same port, the former is 80, the latter is 443.
4. The HTTP connection is simple and stateless; The HTTPS protocol is a network protocol built by the SSL+HTTP protocol for encrypted transmission and authentication, which is more secure than the HTTP protocol.

Crawler Basics---HTTP protocol understanding, Web-based basics, crawler fundamentals