Network Basic Skills Series: The network of those things (updated March 26)

Reprint: Https://community.emc.com/thread/197851?tstart=0

Network basic Skills (eight): Detail TCP sliding window:

Introduction

The distinction between a simple transport protocol such as TCP and UDP is the quality of the data it transmits. TCP tracks the sending of data, and this data management requires that the protocols have the following two key functions:

Reliability : Ensure that the data really arrives at the destination. If it does not arrive, it can be found and re-transmitted.

Data Flow Control : manages the sending rate of data so that the receiving device is not overloaded.

To accomplish these tasks, the entire protocol operation is performed around a sliding window confirmation mechanism . Therefore, the understanding of the sliding window, that is, the understanding of TCP.

More Information

TCP flow-oriented sliding window confirmation mechanism:

TCP treats independent byte data as a stream. It is obviously not feasible to send one byte at a time and receive a confirmation. Even with overlapping transports (that is, sending the next data without waiting for confirmation), the speed is still very slow.

TCP message acknowledgement mechanism as shown above, first, each message has an identification number, each message can be independently confirmed, so at the same time can send more than one message. Device B periodically sends a send restriction parameter to a, which restricts the maximum number of messages that device a can send at a time. Device B can adjust this parameter to control the data flow of device A.

To increase the speed, TCP does not send the data stream as a single byte, but instead divides it into fragments. All the bytes in the fragment are sent and received together, and therefore are confirmed together. The confirmation mechanism does not take the Message ID field, but instead uses the sequence number of the last byte within the fragment. Therefore, you can process different bytes at one time, which is the sequence number within the fragment.

Conceptual classification of TCP data streams

Suppose a new TCP connection is established between A and B. Device a requires a long stream of data to be transmitted, but device B cannot receive all at once, so it restricts device a from sending a specified number of bytes per fragment until the number of bytes sent in the segment is confirmed. After that, device A can continue to send more bytes. Each device tracks the sending, receiving, and confirming data.

If we do a "snapshot" of this process at any point in time, we can divide the data in TCP buffer into the following four classes and treat them as a timeline:

1. the oldest byte in the sent confirmed data stream has been sent and confirmed. This data is viewed from the point of view of the sending device. As shown in the following figure, 31 bytes have been sent and confirmed.

2. bytes that have been sent but have not yet been confirmed sent but not yet confirmed. The sender does not think the data has been processed until it is confirmed. The 14 bytes shown in the following figure are class 2nd.

3. Not sent and the receiving party has not sent the data to the ready device, but the receiver confirms that it has enough space based on the last time the sender has to send a number of bytes. The sender will immediately attempt to send. As shown, the 3rd class has 6 bytes.

4. Not sent and receiver not ready this part of the data is not allowed because the receiver is not ready.

The receiver uses a similar mechanism to differentiate the data that has been received and confirmed, has not yet been accepted but is ready to receive, and has not yet received and is not yet ready to receive. In fact, both the sending and receiving parties maintain a separate set of variables to monitor which type of data is sent and received.

Sequence number settings and synchronization:

The sender and receiver must agree on the sequence number that they will specify for the bytes in the data flow. This process is called synchronization, which is done when the TCP connection is established. To simplify the assumption that the first byte sequence number is 1, the four types of bytes are as follows in the example above:

1. Confirmed bytes 1 to 31 have been sent.

2. Bytes 32 to 45 have been sent but not confirmed.

3. Not sent and the receiver has ready bytes 46 to 51.

4. Not sent and the receiver is not a ready byte 52 to 95.

Send window with available window:

The key operation of the process is the number of unacknowledged bytes that the receiver allows the sender to hold at a time. This is called a send window, sometimes called a window. This window determines the number of bytes allowed to be transmitted by the sender, and is the sum of the bytes of Class 2 and Class 3. Therefore, the dividing line between the last two classes (the receiver is ready but not yet sent and the receiver is not ready) is the addition of a window starting with the first unacknowledged byte. In this example, the first unacknowledged byte is 32, and the entire window size is 20.

The available window is defined as the amount of data that the sender is still allowed to send, given the amount of data being transmitted. is actually equal to the size of class 3rd. The left border is the first byte in the window (byte 32), and the right border is the last byte in the window (byte 51). See below for a detailed explanation of the concept.

TCP class and window size changes after the available window bytes are sent:

When 6 bytes of the third class are immediately sent in the figure above, the 6 bytes are transferred from class 3rd to class 2nd. The byte becomes as follows:

1. Confirmed bytes 1 to 31 have been sent.

2. Bytes 32 to 51 have been sent but not confirmed.

3. Not sent and the receiver has ready bytes of 0.

4. Not sent and the receiver is not a ready byte 52 to 95.

Confirm processing and window scaling:

After a while, the target device sends a confirmation message back to the sender. The target device does not specifically list bytes that it has confirmed, because this results in inefficiency. The maximum number of bytes that the target device sends since the last successful receive .

For example, suppose that the sent unacknowledged bytes (32 to 45) are divided into 4-segment transports: 32-34,35-36,37-41,42-45. Section 1,2,4 has arrived, and 3 paragraphs have not been received. The receiving party will only send back the confirmation information for 32-36. The receiver retains 42-45 but does not acknowledge it because it indicates that the receiver has received 37-41. This is necessary because the TCP acknowledgement mechanism is cumulative and only one number is used to confirm the data. This number is the maximum number of bytes since the last successful receipt. Suppose the target device also sets the window to 20 bytes.

When the sending device receives the acknowledgement, a portion of the 2nd byte is transferred to class 1th because they have been confirmed. Since 5 bytes have been confirmed, the window size has not changed, allowing the sender to send more than 5 bytes. As a result, the window slides 5 bytes to the right. While 5 bytes move from the second class to class 1th, 5 bytes move from class 4th to class 3rd, creating a new available window for the next transport. Therefore, after receiving the confirmation message, it looks like the following figure shows. The byte becomes as follows:

1. Confirmed bytes 1 to 36 have been sent.

2. Bytes 37 to 51 have been sent but not confirmed.

3. Not sent and the receiver has ready bytes from 52 to 56.

4. Not sent and the receiver is not a ready byte 57 to 95.

This process occurs every time the acknowledgment is received, allowing the window to slide across the entire data stream for transmission.

Handling lost acknowledgment information:

But the lost 42-45 how to deal with it. Before the 3rd paragraph (37-41) is received, the receiving device does not send a confirmation message, nor does it send a confirmation of the byte after the paragraph. The sending device can add new bytes to class 3rd, which is 52-56. After sending the device, it stops sending, and the window stays at 37-41.

TCP includes a timing mechanism for transmission and retransmission. TCP will retransmit the missing fragments. One drawback, however, is that because it does not confirm each fragment individually, this can cause other fragments that have actually been received to be re-transmitted (such as 42 to 45).

Report Abuse favorite Show 2 likes (2) 65. Re: Network Basic Skills Series: to elaborate on the network those things (September 29 update) Zhang,jiawen 2014-10-7 5:58 (reply Zhang,jiawen)

Network Basic Skills (ix): elaborate TCP Retransmission

please keep the original source at the beginning of the text: EMC Chinese support forum Https://community.emc.com/go/chinese

Introduction

The main task of TCP is simple: to package and send data. TCP differs from other protocols in that it uses sliding windows to manage the basic data sending and receiving process while ensuring the efficient and reliable transmission of the data stream, so that the sending rate is not significantly faster than the receive rate. This article describes how TCP ensures that the device is reliably and efficiently transmitted. First, the basic methods of detecting lost fragments and retransmission of TCP are described, and then how TCP determines a fragment is lost. More Information

TCP Fragment retransmission timer and retransmission queue:

It is conceptually simple to detect lost fragments and re-transmit them. Each time a fragment is sent, a retransmission timer is turned on. Timers have an initial value and are decremented over time. If the timer times out before the fragment receives confirmation, the fragment is re-transmitted. TCP uses this basic technique, but it is implemented in a slightly different way. The reason is that in order to improve efficiency, multiple unacknowledged fragments need to be processed at once to ensure that each is re-transmitted at the right time. TCP works in the following specific order:

placed in the retransmission queue, a copy of the fragment is placed in a data structure called the retransmission queue , and a retransmission timer is initiated when the timer begins to send a fragment containing the data. Therefore, at some point in time, each fragment is placed in a queue. Queues are arranged according to the remaining time of the retransmission timer, so the TCP software can track those timers in the shortest time period.

Confirmation Processing If a confirmation message is received before the timer times out, the fragment is removed from the retransmission queue.

Retransmission Timeout If no acknowledgement is received before the timer expires, a retransmission timeout occurs and the fragment is automatically re-transmitted. Of course, there is no more guarantee mechanism for retransmission fragments than for the original fragment. Therefore, the fragment remains in the retransmission queue after retransmission. The retransmission timer is restarted and the countdown starts again. If no acknowledgement is received after retransmission, the fragment is re-transmitted and repeated. In some cases, retransmission will also fail. We do not want TCP to be re-transmitted forever, so TCP will only retransmit a certain number of times and determine if a failure terminates the connection.

But how do we know that a fragment is fully confirmed? Retransmission is fragment-based, and TCP acknowledgment information is based on serial number accumulation. Each time when device A sends a fragment to device B, device B views the confirmation Number field for that fragment. All serial numbers below this field have been received by device A. Therefore, when the sequence number of all bytes sent in a fragment is smaller than the last confirmation number of device A to device B, a fragment sent from device B to device A is considered to be confirmed. This is done by calculating the data field of the last sequence number in the fragment combined with the fragment.

Let's take a look at the following diagram to illustrate how confirmation and retransmission work. Assume that the server in the connection has issued four contiguous fragments (numbers starting from 1)

The fragment 1 serial Number field is 1 fragment length 80. So the last sequence number in fragment 1 is 80.

fragment 2 serial number is 81 fragment length is 120. The last sequence number in fragment 2 is 200.

fragment 3 serial number is 201 fragment length is 160. The last sequence number in fragment 3 is 360.

fragment 4 serial number is 361 fragment length is 140. The last sequence number in fragment 3 is 500.

These fragments are sent one after another without having to wait for the previous send to be confirmed. This is a major advantage of the TCP sliding window (in detail the TCP sliding window).

Assuming the client receives the first two transports, it sends back a confirmation message confirming that the number is 201. This tells the server that the first two fragments have been successfully received by the client, and they are removed from the retransmission queue (and the server sends the window 200 bytes to the right). Fragment 3 remains in the retransmission queue before receiving a fragment of confirmation number 361 or higher, and fragment 4 requires a confirmation number of 501 or higher.

Now, let's further assume that fragment 3 was lost during transmission, but fragment 4 was received. The client saves fragment 4 in the receive buffer, but does not need to be acknowledged because TCP is the cumulative acknowledgement mechanism-confirming that fragment 4 indicates that fragment 3 is also received, but not actually. Therefore, the client needs to wait for fragment 3. In fact, the retransmission timer for server-side fragment 3 times out, and the server then retransmissions fragment 3. After the client receives, then sends the acknowledgment information for the fragments 3 and 4 to the server.

Another important question is what the server will do with fragment 4. Although the client is waiting for fragment 3, the server does not receive feedback, so it does not know that fragment 3 is missing, nor does it know what happened to fragment 4 (and the data that is transmitted next). It is possible that the client has received fragment 4 but cannot confirm it, and it is possible that fragment 4 has also been lost. Some implementations will choose to retransmit only fragment 3, and some will retransmit both 3 and 4.

The last issue is the value of the retransmission timer for the fragment used in the retransmit queue. If set too low, excessive retransmission occurs, and if set too high, retransmission of lost fragments can weaken performance. This value must be dynamically adjusted through a process called adaptive retransmission, which is described in the next section.

Report Abuse favorite Show 2 likes (2) 66. Re: Network Basic Skills Series: to elaborate on the network those things (October 8 update) Zhang,jiawen 2014-10-12 7:24 (reply Zhang,jiawen)

Network Basic Skills (10): Elaborate TCP acknowledgement mechanism

please keep the original source at the beginning of the text: EMC Chinese support forum Https://community.emc.com/go/chinese

Introduction

In the TCP acknowledgement mechanism, non-contiguous TCP fragments cannot be effectively processed. The confirmation number indicates that all sequence numbers below this number have been received by the device that sent the number. If the bytes we receive are in the two discontinuous range, it cannot be confirmed by a single number. This can lead to potentially serious performance problems, especially in high-speed or poorly-reliable networks.

More Information

As an example, the server sends 4 fragments and receives 1 replies with a confirmation number of 201. Therefore, fragment 1 and fragment 2 are treated as confirmed. They are removed from the retransmission queue while allowing the server send window to move 200 bytes to the right, sending data by 200 bytes.

However, assuming fragment 3 again, starting with sequence number201, it was lost during the sending process. Because the client has never received this fragment, it cannot send confirmation that the confirmation number is higher than 201, causing the sliding window to stall. The server can continue to send additional fragments until the client's receive window is filled, but until the client sends another acknowledgment, the server's send window does not slide.

Another problem is that if fragment 3 is lost, the client will not be able to tell the server if the subsequent fragments are received. Before the client receive window fills up, it is likely that the client has received fragments 4 and later. However, the client cannot send a confirmation message with a value of 501 to indicate that fragment 4 was received, because this means that fragment 3 is also received .

Here we see the disadvantages of the TCP single number, the cumulative acknowledgement mechanism. We can imagine a worst case scenario where the server is told that it has a 10,000-byte window and 20 fragments per fragment of 500 bytes. The first fragment was lost and the other 19 were received. But since the first fragment was never received, the other 19 could not be confirmed.

The fragment processing policy was not confirmed:

How do we deal with fragments after missing fragments? In this case, when the server Fragment 3 retransmission times out, it must decide what to do with fragment 4, and it does not know whether the client has been received. In the worst case scenario, when the first fragment is lost, the remaining 19 may or may not be received by the client.

There are two possible ways to handle this situation:

Retransmission Timeout fragments only : This is a more conservative way to retransmit only the fragments that have timed out, hoping that the other fragments will be successfully received. If the other fragments after the fragment are actually received, this is the best approach and cannot be performed properly if it is not received. In the latter case, each fragment needs to be timed separately and re-transmitted. Assuming the worst case scenario above, all 20 500-byte fragments are lost. We need to wait for fragment 1 to timeout and retransmit. This fragment may be confirmed, but then we need to wait for fragment 2 to expire and retransmit. This process is repeated several times.

re-transmit all fragments : This is a more radical or more pessimistic way. Whenever a fragment times out, not only will the fragment be re-transmitted, but also all other fragments that have not yet been confirmed. This ensures that there is a waiting time for confirmation at any time, and all unacknowledged fragments are flushed in the event of all unconfirmed fragments being lost, so that the peer device receives an opportunity more than once. In the case where all 20 fragments are lost, a significant amount of time is saved relative to the first one. The problem with this approach is that it is possible that these retransmissions are not necessary. If the first fragment is lost and the other 19 are actually received, the 9500 bytes of data will have to be re-transmitted.

Because TCP does not know whether other fragments are received, it cannot confirm which method is better, but can only choose one way. The example above shows a conservative approach, and the following figure illustrates a radical approach:

The key to the problem is the inability to confirm non-contiguous fragments. The workaround is to extend the TCP sliding window algorithm, adding the ability to allow the device to confirm non-contiguous fragments separately. This feature is called selection confirmation (Selective acknowledgment, SACK).

Select confirmation:

With sack, the connected two-party device must support this feature at the same time, and negotiate whether the sack is allowed through the SYN fragment used to connect. After this process is complete, either device can use the SACK option in a regular TCP fragment. This option contains a list of sequence number ranges for received but unconfirmed fragment data, because they are non-contiguous.

Each device modifies the retransmission queue, and if the fragment has been selected for confirmation, the sack bit position in the fragment is 1. The device uses an improved version of the aggressive approach in Figure 2, after a fragment is re-transmitted, and then all fragments are re-transmitted, unless the sack bit is 1.

For example, in the case of 4 fragments, if the client receives fragment 4 and does not receive fragment 3, when it sends back the acknowledgment with the confirmation number 201 (fragment 1 and Fragment 2), which contains a sack option indicating: "Bytes 361 to 500 have been received but not confirmed". If fragment 4 arrives after Fragments 1 and 2, the above information can also be done with a second confirmation fragment. The server confirms the byte range of fragment 4 and opens the sack bit for fragment 4. When fragment 3 is re-transmitted, the server sees fragment 4 with a sack bit of 1 and does not retransmit it. As shown in the following figure.

After fragment 3 is re-transmitted, the sack bit of fragment 4 is cleared. This is to prevent the client from altering the idea that fragment 4 has received for some reason. The client should send confirmation that the confirmation number is 501 or higher, formally confirming that fragments 3 and 4 are received. If this does not occur, the server must receive another selection confirmation of fragment 4 to open its sack bit, otherwise it will be automatically re-transmitted when the fragment 3 is retransmission or the timer times out.

Network Basic Skills (11): TCP window adjustment and flow control

please keep the original source at the beginning of the text: EMC Chinese support forum Https://community.emc.com/go/chinese

Introduction

The importance of the TCP sliding window size has been described in the previous article. In a client-to-server connection, the client tells the server how many bytes of data it wants to receive from the server at a time, which is the client's receive window, the server's send window. Similarly, the server tells the client how much byte data it wants to receive from the client at a time, that is, the server's receive window and the client's send window.

To understand why window size can fluctuate, first you need to understand its meaning. The simplest way is that it represents the device's receive cache size for a particular connection. That is, the window size represents how much data a device can process from the peer at a time before it is passed on to the application layer for processing.

More Information

When the server receives data from the client, it puts the data in the cache, and the server must do the following two steps for the data:

Confirm : The server must send the acknowledgement back to the client to indicate data reception.

Transport : The server must process the data and pass it to the target application for processing.

It is very important to distinguish between these two things. The key is the basic sliding window mechanism, where the data is acknowledged at reception, but not necessarily transferred from the cache immediately. This means that the cache is likely to be filled when the received data is faster than receiving the TCP processing speed. When this happens, the receiving device needs to resize the window to prevent the cache from being overloaded.

Because of the size of the window to manage the rate at which both ends of the device traffic is connected, TCP is a typical task in this way to achieve flow control in this transport layer. Flow control is important for TCP because it is a way of communicating between devices. By increasing or shrinking the window size, the server and the client can ensure that the data sent to the end is equal to the processing speed.

Reduce The window size to reduce the send rate:

First look at the client-to-server data transfer, as shown in the following figure.

The client transmits 140 bytes of data to the server. After that, the client's available window has 220 bytes left: The 360 bytes of the sending window minus the 140 bytes sent.

After some time, the server receives 140 bytes and places them in the cache. Now, ideally, 140 bytes goes into the cache and is immediately removed from the cache after confirmation. In other words, the cache has enough size to hold all the data sent by the client. The cached free space is maintained at 360 bytes, so the client window is told to remain the same size.

As long as the server processing speed and data entry speed are the same, the window size will remain at 360 bytes. The client moves the 360-byte window 140 bytes to the right after it receives a 140-byte acknowledgment and the window's size remains unchanged. Because the number of bytes is not confirmed now is 0, the client can send 360 bytes of data. Corresponds to the 220 bytes of the previously available window, plus the 140 bytes of data just confirmed.

However, in reality the server may need to handle dozens of, hundreds of, or even thousands of TCP connections. TCP may not be able to process data immediately, or the application itself cannot receive 140 bytes of data. In either case, server TCP cannot immediately remove 140 bytes from the cache. At this point, in addition to sending back the acknowledgement to the client, the server wants to tell the client to change the window size to indicate that the cache has been partially written.

Suppose we receive 140 bytes, but only 40 bytes are sent to the application, and 100 bytes are left in the cache. When sending a 140-byte acknowledgment, the server shrinks the sending window by 100 bytes to 260 bytes. When the client receives this fragment from the server, it will see a 140-byte acknowledgment and slide the window 140 bytes to the right. Reduce the size to 260 bytes during the swipe process. You can think of sliding 140 bytes on the left side of the window, but only 40 bytes on the right side. The new slightly smaller window ensures that the server receives up to 260 bytes of data from the client to accommodate the remaining space in the receive cache, as shown in 1-3 steps in the following figure.

reduce the Send window to stop sending new data:

What if the server is unable to receive any new data? Suppose the client transmits 180 bytes the next time, but the server is too busy to process it. In this case, the server caches the 180 bytes and, in the acknowledgment information, reduces the window size from 260 bytes to 80 bytes. When the client receives a 180-byte acknowledgment, it also sees that the window is reduced by 180 bytes, and it slides the same size as the reduction, informing the server that I am confirming that 180 bytes of data are received, but not allowing you to send new data again. The left side of the window can also be viewed as sliding 180 bytes, but the right side remains motionless. As long as the right side does not move, the client cannot send more data. This process is shown in Figure 4-6 above.

Close the Send window:

Window adjustment can be done by both devices. If the server receives data from the client consistently faster than the rate pushed to the app, the server continues to reduce the receive window. Assuming that the send window is reduced to 80 bytes, the client sends a third request with a length of 80 bytes, but the server is still busy. The server then reduces the window to 0, also known as the close window. This information informs the client that the server is overloaded and needs to stop sending the data completely, as shown in the last step of the previous figure. Then, when the server load is reduced, you can increase the window of the connection again, allowing more data to be transferred.