Go: http protocol details (Classic)

From: http://blog.csdn.net/gueter/archive/2007/03/08/1524447.aspx

Author: Jeffrey

Introduction

HTTP is an object-oriented protocol at the application layer. It is applicable to distributed hypermedia information systems due to its simple and fast method. It proposed in 1990 that, after several years of use and development, it has been continuously improved and expanded. Currently, the sixth version of HTTP/1.0 is used in WWW, standardization of HTTP/1.1 is in progress, and suggestions for HTTP-NG (Next Generation of HTTP) have been put forward.
The main features of HTTP are as follows:
1. Supports the customer/Server mode.
2. simple and fast: when a customer requests a service from the server, they only need to send the request method and path. Common Request methods include get, Head, and post. Each method specifies the type of contact between the customer and the server. Because the HTTP protocol is simple, the program size of the HTTP server is small, so the communication speed is fast.
3. Flexibility: HTTP allows transmission of any type of data objects. The type being transferred is marked by Content-Type.
4. No connection: No connection means that only one request is allowed for each connection. After the server processes the customer's request and receives the customer's response, the connection is disconnected. This method can save transmission time.
5. Stateless: HTTP is stateless. Stateless means that the Protocol has no memory for transaction processing. The lack of status means that if subsequent processing requires the previous information, it must be re-transmitted, which may increase the amount of data transmitted each connection. On the other hand, when the server does not need previous information, its response is faster.

 

I. url for HTTP protocol explanation

HTTP (Hypertext Transfer Protocol) is a stateless, application-layer protocol based on request and response modes. It is often based on TCP connections, http1.1 provides a persistent connection mechanism. Most Web development applications are Web applications built on the HTTP protocol.

The format of http url (a URL is a special type of URI that contains sufficient information for searching a resource) is as follows:
Http: // host [":" port] [abs_path]
HTTP indicates that network resources are to be located through the HTTP protocol; host indicates a valid Internet host domain name or IP address; port specifies a port number. If it is null, the default port 80 is used; abs_path specifies the URI of the requested resource. If abs_path is not provided in the URL, it must be given in the form of "/" when it is used as the request URI. Generally, this work is automatically completed by the browser.
Eg:
1. Enter www.guet.edu.cn
The browser automatically converts to: http://www.guet.edu.cn/
2. http: 192.168.0.116: 8080/index. jsp

 

Ii. http protocol details

An HTTP request consists of three parts: request line, message header, and request body.

1. The request line starts with a method symbol and is separated by spaces, followed by the request URI and Protocol version. The format is as follows: Method Request-Uri http-version CRLF
The method indicates the request method, the request-Uri is a unified resource identifier, the http-version indicates the HTTP protocol version of the request, and the CRLF indicates the carriage return and line feed (except for the CRLF as the end, separate CR or lf characters are not allowed ).

There are multiple request methods (all methods are capitalized). The methods are described as follows:
GET request to get the resource identified by request-Uri
Post attaches new data to the resource identified by request-Uri
Head request to obtain the Response Message Header of the resource identified by request-Uri
The put request server stores a resource and uses request-Uri as its identifier.
The Delete request server deletes the resource identified by request-Uri.
Trace Request information received by the server for testing or diagnosis
Connect reserved for future use
Options requests query server performance, or query resource-related options and requirements
Example:
Get method: when you enter a URL in the address bar of the browser to access the webpage, the browser uses the get method to obtain resources from the server. For example: Get/form.html HTTP/1.1 (CRLF)

The post method requires the request server to accept the data attached to the request. It is often used to submit forms.
Eg: Post/Reg. jsp HTTP/(CRLF)
Accept: image/GIF, image/X-xbit,... (CRLF)
...
HOST: www.guet.edu.cn (CRLF)
Content-Length: 22 (CRLF)
Connection: keep-alive (CRLF)
Cache-control: No-Cache (CRLF)
(CRLF) // This CRLF indicates that the message header has ended and is previously the message header.
User = Jeffrey & Pwd = 1234 // the data submitted below this row

The head method is almost the same as the get method. For the response part of the head request, its HTTP header contains the same information as the GET request. With this method, you do not need to transmit the entire resource content to obtain the information of the resource identified by request-Uri. This method is often used to test the validity, accessibility, and recent updates of hyperlinks.
2. Post-Request Header
3. Request body (omitted)

 

Iii. Response to HTTP protocol details

After receiving and interpreting the request message, the server returns an HTTP Response Message.

HTTP response is composed of three parts: Status line, message header, and response body.
1. The status line format is as follows:
HTTP-version status-code reason-phrase CRLF
HTTP-version indicates the HTTP protocol version of the server, status-code indicates the response status code sent back by the server, and reason-phrase indicates the text description of the status code.
The status code consists of three numbers. The first number defines the response category and has five possible values:
1xx: indicates that the request has been received and continues to be processed.
2XX: Success-indicates that the request has been successfully received, understood, and accepted
3xx: Redirection-further operations are required to complete the request
4xx: client error-the request has a syntax error or the request cannot be implemented
5xx: Server Error -- the server fails to fulfill the valid request
Common status codes, status descriptions, and descriptions:
200 OK // client request successful
400 bad request // The client request has a syntax error and cannot be understood by the server
401 unauthorized // The request is unauthorized. This status code must be used with the WWW-Authenticate header domain
403 Forbidden // The server receives the request but rejects the service.
404 Not found // The requested resource does not exist. For example, the incorrect URL is entered.
500 internal server error // unexpected Server Error
503 server unavailable // The server cannot process client requests currently and may return to normal after a period of time
Eg: HTTP/1.1 200 OK (CRLF)

2. Post-Response Header

3. The response body is the content of the resource returned by the server.

 

Iv. Explanation of HTTP protocol

An HTTP message consists of a client-to-server request and a server-to-client response. Request Message and Response Message are both from the start line (for request message, the start line is the request line, and for response message, the start line is the status line), the message header (optional ), empty line (only CRLF line), message body (optional.

HTTP message headers include common headers, request headers, response headers, and object headers.
Each header field consists of the name + ":" + space + value. The name of the message header field is case-insensitive.

1. Common Header
In a common header, there are a few header fields used for all request and response messages, but not for transmitted entities, only for transmitted messages.
Eg:
Cache-control is used to specify cache commands. cache commands are unidirectional (Cache commands in the response may not appear in the request ), it is independent (the cache command of one message does not affect the cache mechanism of the other message processing), and the similar header domain used by http1.0 is Pragma.
Cache commands for requests include: No-Cache (used to indicate that the request or response message cannot be cached), No-store, Max-age, Max-stale, Min-fresh, only-if-cached;
Cache commands for response include public, private, no-cache, no-store, no-transform, must-revalidate, proxy-revalidate, Max-age, and s-maxage.
For example, to instruct the IE browser (client) Not to cache pages, the Server JSP program can be written as follows: Response. sehheader ("cache-control", "No-Cache ");
// Response. setheader ("Pragma", "No-Cache"); equivalent to the above Code, usually both //
This Code sets the common header domain: cache-control: No-cache in the sent response message.

Date common header field indicates the date and time of message generation

The connection common header field allows sending the specified connection option. For example, if the specified connection is continuous or the "close" option is specified, a notification is sent to the server. After the response is complete, the connection is closed.

2. Request Header
The request header allows the client to send additional request information and client information to the server.
Common request headers
Accept
The accept request header field is used to specify the types of information the client accepts. Eg: accept: image/GIF indicates that the client wants to accept resources in the GIF image format; accept: text/html indicates that the client wants to accept HTML text.
Accept-charset
The accept-charset request header field is used to specify the character set accepted by the client. Eg: Accept-charset: iso-8859-1, gb2312. if this field is not set in the request message, it is acceptable by default for any character set.
Accept-Encoding
The accept-encoding Request Header domain is similar to accept, but it is used to specify acceptable content encoding. Eg: Accept-encoding: gzip. Deflate. If the domain server is not set in the request message, it is assumed that the client can accept all content encoding.
Accept-Language
The accept-language Request Header domain is similar to accept, but it is used to specify a natural language. Eg: Accept-language: ZH-CN. If this header field is not set in the request message, the server assumes that the client is acceptable to all languages.
Authorization
The authorization request header domain is used to prove that the client has the right to view a resource. When a browser accesses a page, if the response code of the server is 401 (unauthorized), it can send a request containing the authorization request header domain, requiring the server to verify the request.
Host (this header field is required when a request is sent)
The host request header field is used to specify the Internet host and port number of the requested resource. It is usually extracted from the http url. For example:
We enter: http://www.guet.edu.cn/index.html in the browser
The request message sent by the Browser contains the host Request Header domain, as follows:
HOST: www.guet.edu.cn
The default port number is 80. If the port number is specified, it is changed to: Host: www.guet.edu.cn: the specified port number.
User-Agent
When we log on to the forum online, we will often see some welcome information, which lists the names and versions of your operating system, the names and versions of your browsers, this is often amazing for many people. In fact, the server application obtains this information from the User-Agent Request Header domain. The User-Agent request header field allows the client to tell the server its operating system, browser, and other attributes. However, this header field is not required. If we write a browser and do not use the User-Agent to request the header field, the server will not be able to know our information.
Example of request header:
GET/form.html HTTP/1.1 (CRLF)
Accept: image/GIF, image/X-xbitmap, image/JPEG, application/X-Shockwave-flash, application/vnd. MS-Excel, application/vnd. MS-PowerPoint, application/MSWord, */* (CRLF)
Accept-language: ZH-CN (CRLF)
Accept-encoding: gzip, deflate (CRLF)
If-modified-since: Wed, 05 Jan 2007 11:21:25 GMT (CRLF)
If-None-Match: W/"80b1a4c018f3c41: 8317" (CRLF)
User-Agent: Mozilla/4.0 (compatible; msie6.0; Windows NT 5.0) (CRLF)
HOST: www.guet.edu.cn (CRLF)
Connection: keep-alive (CRLF)
(CRLF)

3. Response Header
The Response Header allows the server to transmit additional response information that cannot be placed in the status line, as well as information about the server and the next access to the resource identified by the request-Uri.
Common Response Headers
Location
The location response header field is used to redirect the receiver to a new location. Location response header fields are often used when domain names are changed.
Server
The server response header contains the software information used by the server to process requests. It corresponds to the User-Agent Request Header domain. Below is
An example of the server response header domain:
Server: APACHE-Coyote/1.1
WWW-Authenticate
The WWW-authenticate Response Header domain must be included in the 401 (unauthorized) Response Message. When the client receives the 401 Response Message and sends the Authorization Header domain request server to verify the message, the server response header contains this header field.
Eg: www-Authenticate: Basic realm = "basic auth test! "// You can see that the server uses a basic authentication mechanism for requested resources.

4. Object Header
Both request and response messages can be transmitted as an entity. An object consists of the object header domain and the Object Body, but it does not mean that the object header domain and the Object Body must be sent together, but only the object header domain can be sent. The object header defines metadata about the Object Body (eg: whether there is an entity body) and the resource identified by the request.
Common Object Headers
Content-Encoding
The content-encoding object header field is used as a modifier of the media type. Its value indicates the encoding of additional content that has been applied to the Object Body, to obtain the media types referenced in the Content-Type header field, the corresponding decoding mechanism must be adopted. Such as content-encoding, which is used to record the File compression method, eg: Content-encoding: Gzip
Content-language
The content-language object header field describes the natural language used by the resource. If this field is not set, the entity content will be provided to all languages for reading.
. Eg: Content-language: da
Content-Length
The Content-Length object header field is used to specify the length of the Object Body, which is represented by a decimal number stored in bytes.
Content-Type
The Content-Type object header field specifies the media type of the Object Body sent to the recipient. Eg:
Content-Type: text/html; charset = ISO-8859-1
Content-Type: text/html; charset = gb2312
Last-modified
The last-modified object header field is used to indicate the last modification date and time of the resource.
Expires
The expires object header field specifies the response expiration date and time. To enable the proxy server or browser to update the cache after a period of time (when accessing the previously visited page again, load the page directly from the cache, shorten the response time and reduce the server load, we can use the expires object header field to specify the page expiration time. Eg: expires: Thu, 15 Sep 2006 16:23:12 GMT
The client and cache of http1.1 must regard other illegal date formats (including 0) as expired. Eg: to prevent the browser from caching pages, we can also use the expires object header field to set it to 0. The JSP program is as follows: Response. setdateheader ("expires", "0 ");

 

5. Use telnet to observe the communication process of the HTTP protocol

Purpose and principle of the experiment:
Using the MS Telnet tool, you can manually enter the HTTP request information to send a request to the server. After the server receives, interprets, and accepts the request, a response is returned, the response will be displayed in the Telnet window, so as to enhance the understanding of the HTTP communication process from the perceptual aspect.

Tutorial steps:

1. Enable Telnet
1.1 Enable Telnet
Run --> cmd --> Telnet

1.2 Enable telnet echo
Set localecho

2. Connect to the server and send a request
2.1 Open www.guet.edu.cn 80 // note that the port number cannot be omitted

Headers/index. asp HTTP/1.0
HOST: www.guet.edu.cn

/* You can change the Request Method and request the content of the Guilin homepage. Enter the following message */
Open www.guet.edu.cn 80

GET/index. asp HTTP/1.0 // request resource content
HOST: www.guet.edu.cn

2.2 Open www.sina.com.cn 80 // enter Telnet www.sina.com.cn 80 directly under the command prompt symbol
Headers/index. asp HTTP/1.0
HOST: www.sina.com.cn
 

3. Experiment results:

3.1 Request Information 2.1 the response is:

HTTP/1.1 200 OK // request successful
Server: Microsoft-IIS/5.0 // web server
Date: Thu, 08 mar 200707: 17: 51 GMT
Connection: keep-alive
Content-Length: 23330
Content-Type: text/html
Expries: Thu, 08 Mar 2007 07:16:51 GMT
Set-COOKIE: aspsessionidqaqbqqqb = bejcdgkadedjklkkajeoimmh; Path =/
Cache-control: Private

// Resource content omitted

3.2 Request Information 2.2 The response is:

HTTP/1.0 404 Not found // request failed
Date: Thu, 08 Mar 2007 07:50:50 GMT
Server: Apache/2.0.54 <UNIX>
Last-modified: Thu, 30 Nov 2006 11:35:41 GMT
Etag: "6277a-415-e7c76980"
Accept-ranges: bytes
X-powered-by: mod_xlayout_tables/0.0.1vhs.markii.remix
Vary: Accept-Encoding
Content-Type: text/html
X-Cache: Miss from zjm152-78.sina.com.cn
Via: 1.0 zjm152-78.sina.com.cn: 80 <squid/2.6.stables-20061207>
X-Cache: Miss from th-143.sina.com.cn
Connection: Close

Lost connection to the host

Press any key to continue...

4. Note: 1. If an input error occurs, the request will not succeed.
2. the header domain is case-insensitive.
3. For more information about the HTTP protocol, see rfc2616 and find the file at http://www.letf.org/rfc.
4. the development background program must master the HTTP protocol

VI,HTTP-related technical supplements

1. Basics:
High-level protocols include file transfer protocol (FTP), email Transmission Protocol (SMTP), Domain Name System Service (DNS), network news Transmission Protocol (NNTP), and HTTP.
There are three types of mediation: proxy, gateway, and tunnel. A proxy accepts the request according to the absolute format of the URI and overrides all or part of the message, send formatted requests to the server using the uri id. The gateway is a receiving proxy and serves as the upper layer of some other servers. If necessary, you can translate the request to the lower layer server protocol. A channel serves as a relay point between two connections that do not change messages. A channel is often used when communication requires an intermediary (such as a firewall) or an intermediary that cannot identify messages.
Proxy: An intermediate program that can act as a server or a client and create a request for other clients. Requests are transmitted to other servers through possible translation. A proxy must explain before sending the request information and rewrite it if possible. A proxy is often used as a portal through a firewall client. A proxy can also be used as a help application to handle requests that are not completed by a user proxy through the Protocol.
Gateway: a server that acts as an intermediate medium for other servers. Different from the proxy, the gateway accepts the request as if it is the source server for the requested resource; the client sending the request does not realize that it is dealing with the gateway.
The gateway is often used as a portal for servers that use firewalls. The Gateway can also be used as a protocol translator to access resources stored in non-HTTP systems.
Tunnel: it is an intermediary program used as two connection relay. Once activated, the channel is considered not to belong to HTTP Communication, although the channel may be initialized by an HTTP request. When the two ends of the relay connection are closed, the channel disappears. The channel is frequently used when a portal must exist or intermediary cannot interpret the relay communication.

2. Protocol Analysis advantages-HTTP analyzer detects Network Attacks
Analyzing and processing high-level protocols in a modular manner will be the direction of future intrusion detection.
Common ports 80, 3128, and 8080 of HTTP and its proxies are specified using the port label in the network section.

3. HTTP Content lenth restriction vulnerability resulting in DoS Attacks
When using the POST method, you can set contentlenth to define the length of the data to be transmitted, for example, contentlenth: 999999999. Before the transfer is complete, the internal storage will not be released. Attackers can exploit this vulnerability, send junk data to the Web server until the memory of the Web server is exhausted. This attack method basically does not leave any trace.
Http://www.cnpaf.net/Class/HTTP/0532918532667330.html

4. conception of DoS attacks using the characteristics of HTTP
The server is busy processing the attacker's forged TCP connection requests and ignoring the client's normal requests (after all, the client's normal request rate is very small). From the perspective of normal customers, the server loses response, which is called synflood attack (SYN Flood attack) on the server ).
Smurf and Teardrop use ICMP packets to attack flood and IP fragments. This article uses the "normal connection" method to generate DoS attacks.
Port 19 has been used for chargen attacks in the early stage, that is, chargen_denial_of_service,! The method they use is to generate a UDP connection between the two chargen servers so that the server can process too much information and get down. Therefore, there must be two conditions for killing a web server: 1. chargen service 2. HTTP service available
Method: The attacker spoofs the source IP address and sends a connection request (CONNECT) to N chargen servers. After receiving the connection, chargen returns a 72-byte rst stream per second (based on actual network conditions, this is faster) to the server.

5. Http Fingerprint Recognition Technology
The principle of HTTP fingerprint recognition is also the same: records the tiny differences in HTTP protocol execution by different servers. HTTP fingerprint recognition is much more complex than TCP/IP stack fingerprint recognition, because custom HTTP server configuration files, adding plug-ins or components make it easy to change HTTP response information, this makes it difficult to identify; however, the custom TCP/IP stack behavior needs to be modified on the core layer, so it is easy to identify.
It is very easy for the server to return different banner information. For an open-source HTTP server like Apache, users can modify the banner information in the source code, then the HTTP service will take effect again. For HTTP servers that do not have open source code, such as Microsoft's IIS or Netscape, you can modify it in the DLL file that stores banner information, relevant articles have been discussed. I will not go into details here. Of course, this modification is still very effective. another method to blur banner information is to use plug-ins.
Common Test requests:
1: Send basic HTTP requests to head/HTTP/1.0
2: delete/HTTP/1.0 sends unpermitted requests, such as delete requests
3: Get/HTTP/3.0 sends an invalid HTTP Request
4: Get/junk/1.0 sends an incorrect HTTP Request
HTTP fingerprint recognition tool httprint can effectively determine the type of HTTP server by combining fuzzy logic technology with statistical principles. it can be used to collect and analyze signatures generated by different HTTP servers.

6. Others: to improve the performance of your browser, modern browsers also support concurrent access. Multiple connections are established when you browse a Web page, in order to quickly obtain multiple icons on a web page, this can more quickly complete the transmission of the entire web page.
Http1.1 provides this continuous connection method, while the next-generation HTTP protocol: HTTP-NG has increased support for session control, rich content negotiation and other methods to provide
More efficient connection.

Go: http protocol details (Classic)