Complete parsing of an HTTP request response process

The internet is undoubtedly the greatest design in human history, connected with hundreds of millions of of computers and communication devices around the world, even when users on both ends of the globe can communicate in a single instant.

It can be said that "agreement" is to support such a large and complex system of the orderly operation of the core, and the so-called "agreement" is the communication between the two parties must abide by the rules, under such rules, different datagrams may be resolved to different response actions.

In short, "agreement" means that if the sending and receiving parties follow this rule to send data messages, they can get some special functions or services on top of the basic data transmission, otherwise your data is not recognized by others. For example: Follow the TCP protocol at both ends, can be unreliable network transmission of reliable data transmission capability.

The whole computer network is layered, there are seven layers of models, there are five layers of models, individuals feel that five-layer model is more conducive to understanding. We introduce these five layers from top to bottom, which are the application layer, the transport layer, the network layer, the data link layer and the physical layer respectively.

Application Layer

"Application Layer" is the most recent layer from the user, the host on a single process constitutes an "application layer." For example, when you enter "www.baidu.com" in your browser's address bar, what does your browser do on the application layer?

First, the browser will use the DNS protocol to return the IP address of the domain name "www.baidu.com", about the DNS we will explain in detail later.

The application layer then decides to create a "TCP socket" and then encapsulates the request action into an Http datagram and pushes it into the socket.

The socket is divided into two types, "TCP sockets" and "UDP sockets", the application layer may have dozens of datagrams issued, and the transport layer will receive all the response message, then how it can distinguish the message is whose response message?

Sockets are used to distinguish between application-level applications, often by the port number and IP address identification, transport layer as long as the source port and IP address of the response message can be seen to the message to which socket to be pushed.

When an application layer datagram is pushed into the socket, all the work of the application layer is completed, and it is no longer necessary to control the whereabouts of the subsequent messages.

It is also important to note that the "TCP sockets" and "UDP sockets" are essentially the difference between the former to ensure that the datagram reliably arrives at the destination, but inevitably time-consuming, the latter does not guarantee that the datagram will be able to reach the destination, but fast, This is also a point that application layer protocols need to consider when choosing a transport layer protocol.

With regard to TCP and UDP, we will continue to say, let's see how DNS works with domain name Resolution Protocol, and how it will return a domain name to its IP address.

DNS principle

First of all, it is clear that DNS is an application-level protocol, and its choice of Transport layer protocol is UDP, so your domain name resolution process will generally be very fast, but also often appear to resolve the failure of the situation, but refresh a bit better.

On a DNS server, the domain name and its corresponding IP address are stored as a single record, and none of the records can be stored on only one server, and I believe that no matter how powerful a server is, it can't hold hundreds of billions of concurrent traffic around the world.

Roughly speaking, there are three types of DNS servers, root DNS servers, top-level domain DNS servers, and authoritative DNS servers.

Among them, top-level domain DNS servers are primarily responsible for top-level domains such as COM, org, net, edu, gov, and so on.

The root DNS server stores the IP addresses of all the top-level domain DNS servers, which means that you can locate the top-level domain server through the root server. For example: "www.baidu.com", the root server returns all the IP addresses that maintain COM's top-level domain server.

Then you choose one of the top-level domain server, request the top-level domain server, the top-level domain server should be able to make a judgment and give the authoritative server address for the current domain, Baidu, for example, the top-level domain server will return all responsible for Baidu this domain authoritative server address.

So you can choose one of the authoritative server address, to it continue to query "www.baidu.com" specific IP address, the final authoritative server will return to your specific IP address.

At this point, we briefly describe a domain name resolution of the general process, there are some details are not mentioned, we will go through an example to complete the look, the following describes a very important concept.

Throughout the DNS parsing process, there is a very central character we have not introduced it, it is like the host's "assistant", to help the host to query the IP address of the domain name. It is called a "local DNS server".

Every time you get an IP address dynamically through DHCP, this will be said later. In fact, the router not only gives you the IP address, but also tells you a DNS server address, this is your local DNS server address, that is, all your domain name resolution requests just tell it, it will help you check and return the results to you.

In addition, the local DNS server is often a cache function, usually two days of records will be cached, so most of the time you do not feel the domain name parsing process, because often is taken from the cache, very fast.

Let's look at a simple case:

Online looking for a picture, your own painting is too time-consuming, but enough to illustrate the problem, now assume the request "www.xx.com".

• ①: The host sends a query message to the local DNS responsible for itself, and if there is a local server cache, the result will be returned directly
• ②: The local server discovery cache is not in, so from the built-in internal root server list Select a Send query message
• ③: root server resolves the suffix name and tells The local server that it is responsible for all top-level server lists for. com
• ④: The local server selects a top-level domain server to continue the query, and the. com domain server continues to parse after receiving the domain name, returning the responsible. xx domain List of all authoritative servers
• ⑥: The local server sends the query message again from one of the returned authoritative servers, eventually obtaining a specific IP address from one of the authoritative servers
• ⑧: Returning results to the host

In fact, the entire DNS message delivery and response process is to go to our five layer protocol, but the focus here is to understand the DNS protocol itself, so did not mention the other layers of specific details, the emphasis here is to remind you that DNS is only an application layer protocol.

Transport Layer

The task of the transport layer is to collect all the datagrams for the application layer to eject the socket, and to re-encapsulate the application layer datagram according to the Transport layer protocol specified by the application layer, and then push it to the network layer for delivery.

TCP and UDP are the transport layer of the two protocol, the former is a reliable connection-based transport protocol, the latter is a non-connected unreliable transport protocol, so the former is more suitable for some high data integrity requirements, the latter is suitable for the kind of data can be lost but very high transmission rate requirements of the scene, such as: voice telephone, Video, and so on, lost one or two bags up to a bit, harmless.

Udp

UDP is not as complex as TCP, it does not guarantee the reliable transmission of data to the destination, and does not guarantee that the data arrive at the destination sequentially, simply provides a simple error test. The message format is as follows:

The data is the application layer of data, the source port number for the delivery of response messages, the destination port number is used to deliver data to the destination process, checksum to check the transmission of data is damaged, if damaged, UDP will directly discard the message.

Tcp

TCP is a bit more complex, it is connection-oriented, and provides a reliable data transmission service based on the connection, and its packet format is as follows:

Simply explaining the meaning of the various fields in the message format is not too practical, and you can hardly understand how TCP is "three-time handshake", "four Waves" and "lost packet retransmission," and so on, how to use the relevant fields in the message when the action is not interrupted.

First of all, we look at the familiar "three-time handshake", which is basically synonymous with TCP, no matter understand the specific principle of the person, refers to TCP, basically know "three times handshake."

In itself, thethree-time handshake of TCP is to ensure that both sides of the communication can establish a stable connection and complete the request and response action of the data message , as to why the three-time handshake instead of four five times, this is a philosophical issue, this is not discussed here.

The first step:

The client sends a special TCP message to the server that does not contain the application layer data and is a special message with a SYN field value of 1 in the TCP header (see the message format above).

In addition, the client will randomly generate an initial sequence number, filled in the message "ordinal" field, representing the current message sequence number is this, and my subsequent groupings will be based on this ordinal increment.

The message will then be sent to the server via the network layer, link layer, and physical layer.

Step Two:

If the packet is lost, the client attempts to send again at some time interval.

And if the packet arrives at the server exactly, the server disconnects the TCP header and sees that it is a special SYN handshake message that allocates the cache and other resources for this connection.

Then the server begins to build the response message, and the SYN is a field to be used for synchronization, and the response message will still be set to 1, and the server will randomly generate an ordinal field of the response message placed in the initial sequence number.

Finally, the service side also assigns a value to the Acknowledgment field in the response message, which is the value of the ordinal that the client sends over.

The overall meaning is, "I agree with your connection request, my initial serial number is XXX, your initial serial number I received, I wait for your next group to arrive"

Step Three:

The client receives a response message from the server and allocates resources such as the cache necessary for the client TCP connection, and the connection is established.

In fact, from the third step, the client can carry the application layer data to the server to exchange messages, in each subsequent message, the SYN is 0, because it is only used to synchronize the initial sequence number, this needs to be clear.

Overall, the entire "handshake" process is roughly as follows:

Let's look at the process of removing the "four waves" of a TCP connection.

Because a TCP connection consumes a large amount of host resources, not only the server needs to allocate a variety of cache resources, the client also needs to allocate the appropriate resources. Because TCP is "full-duplex communication," the server and the client are actually the same, and who are the clients who are the servers are relative.

It is important to note that it is clear that a TCP connection is not only broken by the client, but also the server can be actively disconnected.

Here we assume that the client is actively initiating a disconnected request as an example:

The first step:

The client constructs a special TCP message, the header field of which FIN is set to 1, and the message is sent.

Step Two:

The service side receives the special FIN message, then responds to the client an ACK message, tells the client that the request shutdown message has been received, I am processing.

Step Three:

The server sends a FIN message that tells the client that I'm going to close the connection.

Fourth Step:

The client returns an ACK response message, tells the server that I received the message you just sent, and I have confirmed that you can close the connection.

When the server receives an ACK response message sent by the client, all the resources used by the server for that TCP connection are freed, and the client periodically waits for a certain amount of time to fully release its associated resources for that connection.

A more intuitive description of a picture:

Together with the graph and related serial number information, we will talk about some of the details.

First, the client sends a special grouping with the sequence number U. After sending, the client enters the Fin-wait-1 state, in which case the TCP connected client can no longer send datagrams, but it is acceptable for the datagram, and it waits for the response message from the server.

Then, the server receives the client sends the termination connection message request, the service side constructs the response message, tells the client "serial number u+1 before the grouping I have received", and enters the close-wait state, this state duration is very short.

The service side will send its FIN datagram immediately, notifying the client that my server is about to close the connection, and then enters the Last_ack state to wait for the client response message.

Once the client receives this FIN message, it returns a confirmation message and enters the time-wait state, waiting for the 2MSL time interval to fully release the resource occupied by the client TCP connection.

At the same time, when the server receives the final acknowledgement from the client, it will disconnect the server and release the related resources directly.

As to why the last client needs to wait 2MSL time to completely release TCP-related resources?

That is because 2MSL is a message is stored in the network for the longest time, more than the time to arrive at the message will be discarded, and if the client's last acknowledgment message is lost in the network, the server will initiate a time-out request, resend the third wave action, the waiting client can then resend a confirmation request.

This is the reason why the client waits for a maximum message transfer time. One might wonder why the previous requests did not do a timeout wait and only the last data send did a timeout wait?

In fact, the reason is very simple, I believe you can also think of, is the TCP self-timing ability, more than a certain period of time without receiving a message confirmation message, will automatically resend, and here if you do not wait to directly close the connection, then how do I know the server has not received my confirmation message.

By waiting for a maximum period, if this period does not receive the service side of the message request, then our confirmation message must have reached the service side, or repeat the send once.

At this point, TCP's "three-time handshake" and "four waves" we have simply described the completion, below we look at some other TCP features, such as: reliable transmission, congestion control, etc.

First, we look at how TCP realizes reliable transmission, that is, how to solve the problem of packet loss in network transmission.

TCP uses the "Fallback N-step" protocol for reliable transmission, and, to be exact, TCP is part of the optimization based on it.

The "Fallback N-step" protocol is also referred to as the "sliding window" protocol, which allows the sender to have n "sent but not confirmed" data packets, p1 to P3 length is n, where the window refers to P1 to P3 this interval.

Only when the sender receives a P1 confirmation message can the entire window slide forward, and the server will only cache these "unexpected messages" until the P1 acknowledgement is received, even if the message behind it has been received.

Until the server receives the minimum expected message, the incoming message is removed from the cache, merged and delivered upwards, and a confirmation message is returned to the sender.

When the sending window has been confirmed by multiple messages from left to right, the entire window will slide forward multiple unit lengths.

Let's look at an example:

This is a sender's window, gray indicates that the message has been confirmed, yellow indicates the message has been sent but not confirmed, Green indicates the next message to be sent, white indicates the message is not available.

This is the assumption that the server has received 6 or 72 messages, but it was last submitted to the application layer is the 4th message, that is, it is waiting for the 5th message, so it will temporarily cache 6, 72 messages, until the 5th message is delivered to the application layer.

Now the 5th message due to time-out was re-transmitted, and finally arrived at the destination, the service ended up delivering 5, 6, 73 copies of the message, and returned a confirmation message, ACK = 8, indicating that all messages before the ordinal 8 were received .

When the sender receives the confirmation message, 5, 6, 7 become gray, the window moves forward three units of length.

In addition, I would like to highlight a detail, TCP is no negative confirmation, so if the service side of the continuous response of the multiple message is the same serial number confirmation, it is likely that the serial number after a message is lost.

For example, if the server sends multiple ACK acknowledgments for Group 5, what does it say? Description of the current service end of the complete delivery of the serial number is 5th, the subsequent message I did not receive, you'd better send it again don't wait for the timeout.

This is also the core principle of "fast retransmission".

So the reliable transmission of TCP We also basically finished, below we see if the network congestion, TCP is how to control the sending traffic?

TCP thinks: Packet loss is congestion, need to reduce the transmission efficiency, and every time you receive a confirmation datagram that the network unobstructed, will increase the efficiency of the transmission.

TCP Congestion control algorithm consists of three parts, slow start, congestion avoidance and fast recovery .

The idea of slow start is that it starts with a slow send, such as sending a datagram only once in a certain time period, and when the confirmation message is received, the next time interval will be twice times the rate of two datagrams sent, and so on.

So, for a short period of time, the sender of a TCP connection will grow exponentially, but once a packet loss occurs, either a redundant ACK acknowledgement is received, or a packet acknowledgment ACK is never received and a time-out retransmission has to be initiated, the sender considers the "network is congested".

The rate is then adjusted directly to one, that is, a round trip time period, only one packet is sent, and a variable ssthresh is set to express the concept of a threshold, which is half the sender's sending rate at the time of the last packet loss.

After the sender's sending efficiency will increase exponentially, but unlike the first time, once this threshold is reached, TCP will enter the "congestion avoidance" mode, the mode of transmission efficiency will no longer exponential growth, will be cautious growth.

The idea of congestion avoidance is that when all datagrams sent in each round-trip period are confirmed, the next time a packet is sent, the slower growth efficiency is prudent.

So once the send-side timeout drops, note that this is the time-out, the sending rate is set to re-enter the slow boot state, the threshold is half the current transmission efficiency.

And if the server returns multiple redundant ACK to clarify your packet loss, TCP thinks this is not serious, in this case, TCP halved the current delivery efficiency and goes into the fast recovery phase.

The basic idea of rapid recovery is to receive several redundant ACK to increase the delivery efficiency of several packets, that is, you do not receive a few messages from my server, these two send me the rate of increase quickly sent to you.

When this occurs during the packet loss caused by the send side timeout, the same processing mode, the initialization of the sending rate to halve the current transmit efficiency as a threshold, into the slow start phase.

Of course, if you receive a confirmation of the lost message during this period, then the transmission efficiency will be reduced and the congestion avoidance state should be reached.

In this way, the entire TCP core of several ideas have been introduced, the entire transport layer is basically clear. You should have a certain understanding of the transport layer, and I'll summarize it.

The task of the transport layer is to retrieve all the data that needs to be sent from the sockets of each process in the application layer, then select TCP or UDP to encapsulate the data and push it to the network layer below to be sent.

Complete parsing of an HTTP request response process