Deep understanding of HTTP protocol-http protocol (really classic)

Introduction

HTTP is an object-oriented protocol belonging to the application layer, which is suitable for distributed hypermedia information System because of its simple and fast way. It was proposed in 1990, after several years of use and development, has been continuously improved and expanded. Currently used in the WWW is the sixth edition of Http/1.0, http/1.1 standardization work is in progress, and Http-ng (Next Generation of HTTP) has been proposed.
The main features of the HTTP protocol can be summarized as follows:
1. Support client/server mode.
2. Simple and fast: When a customer requests a service from the server, it simply transmits the request method and path. The request method commonly has, POST. Each method specifies a different type of contact between the customer and the server. Because the HTTP protocol is simple, the HTTP server's program size is small, so the communication speed is fast.
3. Flexible: HTTP allows the transfer of any type of data object. The type being transmitted is marked by Content-type.
4. No connection: The meaning of no connection is to limit the processing of only one request per connection. When the server finishes processing the customer's request and receives the customer's answer, the connection is disconnected. In this way, the transmission time can be saved.
5. Stateless: The HTTP protocol is a stateless protocol. Stateless means that the protocol has no memory capacity for transactional processing. A lack of state means that if the previous information is required for subsequent processing, it must be re-routed, which may cause the amount of data to be transferred per connection to increase. On the other hand, it responds faster when the server does not need the previous information.

First, the HTTP protocol detailed URL Chapter

HTTP (Hypertext Transfer Protocol) is a request-and-response mode-based, stateless, application-level protocol, often based on TCP connection, HTTP1.1 version of a continuous connection mechanism, the vast majority of web development, is built on the HTTP protocol on the Web application.

The HTTP URL (url is a special type of URI that contains enough information to find a resource) in the following format:
http://host[":" Port][abs_path]
HTTP means to locate network resources through the HTTP protocol, the host represents a legitimate Internet host domain name or IP address; port specifies a port number, or null specifies the URI of the requested resource using the default port 80;abs_path, if no abs_ is given in the URL Path, it must be given as a "/" when it is a request URI, which is usually done automatically by the working browser.
eg
1. Input: www.guet.edu.cn
Browser automatically converted to: http://www.guet.edu.cn/
2, http:192.168.0.116:8080/index.jsp

Second, the HTTP protocol detailed request chapter

The HTTP request consists of three parts: the request line, the message header, the request body

1. The request line begins with a method symbol, separated by a space, followed by the requested URI and version of the Protocol, in the following format: Method Request-uri http-version CRLF
Where method means the request, Request-uri is a Uniform Resource identifier, http-version represents the HTTP protocol version of the request, CRLF indicates carriage return and newline (except for the CRLF at the end, a separate CR or LF character is not allowed).

There are several ways to request a method (all uppercase), and each method is interpreted as follows:
Get request gets the resource identified by the Request-uri
Post appends new data to the resource identified by Request-uri
HEAD request Gets the response message header for the resource identified by Request-uri
PUT Request server stores a resource and uses Request-uri as its identity
Delete Request server deletes the resource identified by the Request-uri
TRACE requests the server to echo received request information, primarily for testing or diagnostics
CONNECT reserved for future use
Options request the performance of the query server, or query for resource-related choices and requirements
Application Examples:
Get method: When you access a Web page by entering a URL in the address bar of the browser, the browser uses the Get method to get resources to the server, eg:get/form.html http/1.1 (CRLF)

The Post method requires the requested server to accept the data appended to the request and is often used to submit the form.
eg:post/reg.jsp http/(CRLF)
Accept:image/gif,image/x-xbit,... (CRLF)
...
HOST:www.guet.edu.cn (CRLF)
Content-length:22 (CRLF)
Connection:keep-alive (CRLF)
Cache-control:no-cache (CRLF)
(CRLF)//The CRLF indicates that the message header has ended, preceded by a message header
user=jeffrey&pwd=1234//The following line is the submitted data

The head method is almost the same as the Get method, and for the response part of the head request, the information contained in the HTTP header is the same as the information obtained through the GET request. Using this method, you can obtain information about the resources identified by Request-uri without transmitting the entire resource content. This method is commonly used to test the validity of hyperlinks, whether they can be accessed, and whether they have been updated recently.
2. After the request header is described
3. Request body (slightly)

Third, the HTTP protocol detailed response Chapter

After receiving and interpreting the request message, the server returns an HTTP response message.

The HTTP response is also made up of three parts: status line, message header, response body
1, the status line format is as follows:
Http-version Status-code reason-phrase CRLF
Where http-version represents the version of the server HTTP protocol, Status-code represents the response status code sent back by the server, and Reason-phrase represents a textual description of the status code.
The status code consists of three digits, the first number defines the category of the response, and there are five possible values:
1XX: Indication information--Indicates that the request has been received and continues processing
2XX: Success-Indicates that the request has been successfully received, understood, accepted
3XX: Redirect--further action is required to complete the request
4XX: Client Error--Request syntax error or request not implemented
5XX: Server-side error-the server failed to implement a legitimate request
Common status codes, status descriptions, descriptions:
$ OK//client request succeeded
Bad Request//client requests have syntax errors and cannot be understood by the server
401 Unauthorized//request unauthorized, this status code must be used with the Www-authenticate header field
403 Forbidden//server receives request, but refuses to provide service
404 Not Found//request resource not present, eg: Wrong URL entered
Internal Server error//server unexpected errors
503 Server Unavailable//server is currently unable to process client requests and may return to normal after some time
eg:http/1.1 OK (CRLF)

2. The response header is described later

3, the response body is the contents of the resources returned by the server

Iv. HTTP protocol Details of the message header chapter

HTTP messages consist of client-to-server requests and server-to-client responses. Both the request message and the response message are from the start line (for the request message, the start line is the request line, for the response message, the start line is the status line), the message header (optional), the empty line (only the CRLF line), and the message body (optional) is composed.

The HTTP message header includes the normal header, the request header, the response header, and the entity header.
Each header field consists of a name + ":" + a Space + value, and the name of the message header field is case-insensitive.

1. Normal header
In the normal header, a small number of header fields are used for all request and response messages, but not for the transferred entity, only for the transmitted messages.
eg
The Cache-control is used to specify the cache instruction, the cache instruction is unidirectional (the cache instruction appearing in the response may not appear in the request), and is independent (the cache instruction of one message does not affect the caching mechanism of another message processing), and HTTP1.0 uses a similar header domain of pragma.
Cache directives at request include: No-cache (used to indicate that the request or response message cannot be cached), No-store, Max-age, Max-stale, Min-fresh, only-if-cached;
Cache directives for response include: public, Private, No-cache, No-store, No-transform, Must-revalidate, Proxy-revalidate, Max-age, S-maxage.
Eg: in order to instruct IE browser (client) not to cache the page, the server-side JSP program can be written as follows: Response.sehheader ("Cache-control", "No-cache");
Response.setheader ("Pragma", "No-cache"), function equivalent to the above code, usually both//shared
This code will set the normal header field in the Sent response message: Cache-control:no-cache

Date Normal header field indicates the day and time the message was generated

The connection normal header field allows the option to send a specified connection. For example, specify that the connection is contiguous, or specify a "close" option to notify the server to close the connection after the response is complete

2. Request Header
The request header allows the client to pass additional information about the request to the server side, as well as the client itself.
Common Request Headers
Accept
The Accept Request header field is used to specify which types of information the client accepts. Eg:accept:image/gif, indicating that the client wants to accept a resource in GIF image format; accept:text/html, indicating that the client wants to accept HTML text.
Accept-charset
The Accept-charset request header field is used to specify the character set accepted by the client. eg:accept-charset:iso-8859-1,gb2312. If the field is not set in the request message, the default is to accept any character set.
Accept-encoding
The Accept-encoding request header field is similar to accept, but it is used to specify acceptable content encoding. Eg:accept-encoding:gzip.deflate. If the domain server is not set in the request message, the client is assumed to be acceptable for various content encodings.
Accept-language
The Accept-language request header field is similar to accept, but it is used to specify a natural language. EG:ACCEPT-LANGUAGE:ZH-CN. If the header field is not set in the request message, the server assumes that the client is acceptable for each language.
Authorization
The authorization request header domain is primarily used to prove that a client has permission to view a resource. When a browser accesses a page, if a response code of 401 (unauthorized) is received from the server, a request containing the authorization request header domain can be sent, requiring the server to validate it.
Host (the header field is required when the request is sent)
The host request header domain is primarily used to specify the Internet host and port number of the requested resource, which is usually extracted from the HTTP URL, eg:
We enter in the browser: http://www.guet.edu.cn/index.html
In the request message sent by the browser, the host Request header field is included, as follows:
Host:www.guet.edu.cn
The default port number 80 is used here, and if a port number is specified, it becomes: Host:www.guet.edu.cn: Specify port number
User-agent
When we go online to the forum, often see some welcome information, which lists the name and version of your operating system, the name and version of the browser you are using, which often makes a lot of people feel amazing, in fact, the server application is from user-agent this request header domain to obtain this information. The User-agent request header domain allows the client to tell the server about its operating system, browser, and other properties. However, this header field is not required, and if we write a browser ourselves without using the User-agent request header domain, then the server side will not be able to know our information.
An example of a request header:
Get/form.html http/1.1 (CRLF)
Accept:image/gif,image/x-xbitmap,image/jpeg,application/x-shockwave-flash,application/vnd.ms-excel,application /vnd.ms-powerpoint,application/msword,*/* (CRLF)
ACCEPT-LANGUAGE:ZH-CN (CRLF)
Accept-encoding:gzip,deflate (CRLF)
if-modified-since:wed,05 Jan 11:21:25 GMT (CRLF)
if-none-match:w/"80b1a4c018f3c41:8317" (CRLF)
user-agent:mozilla/4.0 (compatible; MSIE6.0; Windows NT 5.0) (CRLF)
Host:www.guet.edu.cn (CRLF)
Connection:keep-alive (CRLF)
(CRLF)

3. Response header
The response header allows the server to pass additional response information that cannot be placed in the status line, as well as information about the server and the next access to the resources identified by Request-uri.
Common response Headers
Location
The Location response header field is used to redirect the recipient to a new position. Location response header fields are commonly used when changing domain names.
Server
The server Response header field contains the software information that the server uses to process the request. Corresponds to the User-agent request header field. Below is
An example of the server Response header field:
server:apache-coyote/1.1
Www-authenticate
The www-authenticate response header domain must be included in the 401 (unauthorized) response message, the client receives a 401 response message, and when the authorization header domain is sent to the request server to validate it, the service-side response header contains the header domain.
Eg:www-authenticate:basic realm= "Basic Auth test!" You can see that the server is using a Basic authentication mechanism for the requested resource.

4. Entity Header
Both request and response messages can send an entity. An entity consists of an Entity header field and an entity body, but it does not mean that the entity header fields and entity bodies are sent together, and only the entity header fields can be sent. The entity header defines the meta-information about the entity body (eg: there is no entity body) and the resource identified by the request.
Common entity Headers
Content-encoding
The Content-encoding Entity header field is used as a modifier for the media type, and its value indicates the encoding of additional content that has already been applied to the entity body, so the corresponding decoding mechanism must be used to obtain the media type referenced in the Content-type header domain. Content-encoding This method of compressing the document, Eg:content-encoding:gzip
Content-language
The Content-language Entity header field describes the natural language used by the resource. The domain is not set and the entity content is considered to be available to all languages for reading
Stakeholders Eg:content-language:da
Content-length
The Content-length Entity header field is used to indicate the length of the entity body, expressed as a decimal number stored in bytes.
Content-type
The Content-type Entity header field term indicates the media type that is sent to the recipient's entity body. eg
Content-type:text/html;charset=iso-8859-1
content-type:text/html;charset=gb2312
Last-modified
The Last-modified Entity header field is used to indicate the last modification date and time of the resource.
Expires
The Expires Entity header field gives the date and time when the response expires. In order for a proxy server or browser to update the cache after a period of time (once again accessing pages that have been visited, loading directly from the cache, shortening response times, and reducing server load), we can use the Expires entity header domain to specify when the page expires. eg:expires:thu,15 SEP 2006 16:23:12 GMT
The HTTP1.1 client and cache must treat other illegal date formats (including 0) as expired. Eg: in order to let the browser do not cache the page, we can also take advantage of the Expires entity header domain, set as 0,jsp in the program as follows: Response.setdateheader ("Expires", "0");

V. Using TELNET to observe the communication process of the HTTP protocol

Experimental purpose and Principle:
Using MS's Telnet tool, a request is made to the server by entering the HTTP request information manually, and the server receives, interprets, and accepts the request and returns a response that is displayed on the Telnet window to deepen the perception of the HTTP protocol's communication process.

Experimental steps:

1. Turn on Telnet
1.1 Turning on Telnet
Run-->cmd-->telnet

1.2 Turning on the Telnet echo feature
Set Localecho

2. Connect to the server and send the request
2.1 Open www.guet.edu.cn 80//Note the port number cannot be omitted

Head/index.asp http/1.0
Host:www.guet.edu.cn

/* We can change the request method, request Guilin Electronic homepage content, enter the message as follows */
Open www.guet.edu.cn 80

get/index.asp http/1.0//content of request resources
Host:www.guet.edu.cn

2.2 Open www.sina.com.cn 80//Enter Telnet www.sina.com.cn directly under the command prompt symbol 80
Head/index.asp http/1.0
Host:www.sina.com.cn

3 Experimental results:

3.1 Request Information 2.1 The resulting response is:

http/1.1 ok                                               //Request Success
server:microsoft-iis/5.0                                     //web Server
date:thu,08 Mar 200707:17:51 GMT
connection:keep-alive                                  &NBSP
content-length:23330
content-type:text/html
expries:thu,08 Mar 07:16:51 GMT
set-cookie:aspsessionidqaqbqqqb=bejcdgkadedjklkkajeoimmh; path=/
Cache-control:private

Omission of resource contents

3.2 Request Information 2.2 The resulting response is:

http/1.0 404 Not Found//request failed
Date:thu, Mar 07:50:50 GMT
server:apache/2.0.54 <Unix>
Last-modified:thu, 2006 11:35:41 GMT
ETag: "6277a-415-e7c76980"
Accept-ranges:bytes
X-powered-by:mod_xlayout_jh/0.0.1vhs.markii.remix
Vary:accept-encoding
Content-type:text/html
X-cache:miss from zjm152-78.sina.com.cn
via:1.0 zjm152-78.sina.com.cn:80<squid/2.6.stables-20061207>
X-cache:miss from th-143.sina.com.cn
Connection:close

I lost the connection to the mainframe.

Press any key to continue ...

4. Note: 1, an input error occurred, the request will not succeed.
2, the header field is not case-sensitive.
3, a deeper understanding of the HTTP protocol, you can view RFC2616, find the file on the HTTP://WWW.LETF.ORG/RFC.
4, the development background program must master the HTTP protocol

Six, HTTP Protocol Related Technical supplement

1. Foundation:
High-level protocols include: File Transfer Protocol FTP, e-Mail Transfer Protocol SMTP, Domain Name System service DNS, Network News Transfer Protocol NNTP and HTTP protocol, etc.
Mediation consists of three types: proxy, gateway, and channel (tunnel), an agent accepts requests based on the absolute format of the URI, rewrites all or part of the message, and sends the formatted request to the server through the identity of the URI. The gateway is a receiving agent that acts as the upper layer of some other servers and, if necessary, translates the request to the underlying server protocol. A channel acts as a relay point between two connections that do not change the message. The channel is often used when the communication needs to pass through an intermediary (for example, a firewall, etc.) or if the content of the message is not recognized by the intermediary.
Proxy: An intermediary program that can act as a server or as a client to establish requests for other clients. Requests are either internally or passed to other servers through possible translations. An agent must interpret and overwrite it if possible before sending the request information. Proxies are often used as portals through the firewall's client side, and proxies can be used as a help app to handle requests that are not completed by the user agent through the protocol.
Gateway: A server that acts as an intermediary for other servers. Unlike the proxy, the gateway accepts the request as if it were the source server for the requested resource, and the requesting client is unaware that it is dealing with the gateway.
Gateways are often used as server-side portals through firewalls, and gateways can be used as a protocol translator to access resources stored in non-HTTP systems.
Channel (tunnel): is a broker that is a relay of two connections. Once activated, the channel is considered not to be an HTTP communication, although the channel may be initialized by an HTTP request. The channel disappears when both ends of the relayed connection are closed. A channel is often used when a portal must exist or the intermediary (intermediary) cannot interpret the relay's traffic.

2. Advantages of Protocol Analysis-http Analyzer detects network attacks
The analysis and processing of high-level protocols in a modular manner will be the direction of future intrusion detection.
Common ports 80, 3128, and 8080 for HTTP and its proxies are specified in the network section with the port tag.

3. The HTTP protocol content lenth limit vulnerability causes a denial of service attack
When using the Post method, you can set Contentlenth to define the length of the data that needs to be transferred, such as contentlenth:999999999, which is not released until the transfer is complete, and the attacker can exploit this flaw Continuously sends spam data to the Web server until the Web server runs out of memory. This method of attack does not leave a trace.
Http://www.cnpaf.net/Class/HTTP/0532918532667330.html

4. Some ideas for denial-of-service attacks using the features of the HTTP protocol
The server is busy processing an attacker's bogus TCP connection request, ignoring the customer's normal request (after all, the client's normal request rate is very small), at this point, from the normal customer's point of view, the server is unresponsive, this situation we call: The server side was Synflood attack (SYN flood attack).
Smurf, teardrop and so on are using ICMP packets to flood and IP fragment attacks. This article uses a "normal connection" method to generate a denial of service attack.
19 ports in the early days already someone used to do chargen attacks, namely Chargen_denial_of_service, but! They use the method is to create a UDP connection between the two Chargen servers, so that the server processing too much information and down, then, to kill a Web server must have 2 conditions: 1. Chargen Service 2. HTTP Service
Method: The attacker forged the source IP to send a connection request (connect) to n Chargen, and after receiving the connection, the Chargen will return a stream of 72 bytes per second (in fact, this speed is faster than the actual network) to the server.

5. HTTP Fingerprint recognition technology
The principle of HTTP fingerprint recognition is basically the same: record different servers to identify the minor differences in HTTP protocol execution. HTTP fingerprinting is much more complex than TCP/IP stack fingerprinting, because of custom HTTP server configuration files, Adding plug-ins or components makes it easy to change the response of HTTP, which makes identification difficult, whereas customizing the TCP/IP stack requires modifying the core layer so it is easy to identify.
To make the server return different banner information settings is very simple, such as Apache, open source HTTP server, the user can modify the banner information in the source code, and then restart the HTTP service to take effect For HTTP servers that do not have public source code, such as Microsoft IIS or Netscape, can be modified in the DLL file where the banner information is stored, the relevant articles are discussed, and are not described here. Of course, the effect of such a modification is good. Another way to blur banner information is to use a plugin.
Common Test requests:
1:head/http/1.0 sending a basic Http request
2:delete/http/1.0 send requests that are not allowed, such as DELETE requests
3:get/http/3.0 sending an illegal version of the HTTP protocol request
4:get/junk/1.0 sending an incorrect specification of an HTTP protocol request
HTTP fingerprint Identification Tool Httprint, it can effectively determine the type of HTTP server by using the principle of statistics and combining fuzzy logic technology. It can be used to collect and analyze signatures generated by different HTTP servers.

6, Other: In order to improve the user's performance when using the browser, the modern browser also supports concurrent access, browse a Web page while establishing multiple connections, to quickly obtain a number of icons on a Web page, so that the entire page can be faster to complete the transmission.
This continuous connection is provided in HTTP1.1, while the next-generation HTTP protocol: Http-ng adds support for session control, rich content negotiation, and more to provide
more efficient connections.

Deep understanding of HTTP protocol-http protocol (really classic)