Efficiency of small data packets transmitted over TCP

Abstract: when using TCP to transmit small data packets, the program design is very important. If the TCP packet is not configured in the design scheme
Latency Response, Nagle algorithm, and Winsock buffering will seriously affect program performance. This article discusses these
This section lists two cases and provides some optimization solutions for transmitting small data packets.

Background: When the Microsoft TCP stack receives a packet, a 200-millisecond timer is started. When ACK confirms data packets
After the timer is sent, the timer is reset. When the next packet is received, the timer of 200 milliseconds is started again. To improve applications
In terms of Intranet and Internet transmission performance, Microsoft TCP stack uses the following policies to determine the transmission performance after packets are received.
When to send ACK validation packets:
1. If the next packet is received before the 200 millisecond timer times out, an ACK confirmation packet is sent immediately.
2. If there is a packet that needs to be sent to the receiving end of the ACK confirmation information, the ACK confirmation information is included in the packet and sent immediately.
3. When the timer times out, the ACK confirmation message is sent immediately.
To avoid the network congestion of small data packets, the Nagle algorithm is enabled by default in Microsoft TCP stack.
Call Send to concatenate the sent data and Send it together when the ACK confirmation message of the previous data packet is received. Below is the Nagle
Exceptions of algorithms:
1. If the data packet spliced by the Microsoft TCP stack exceeds the MTU value, the data will be sent immediately without waiting for the previous data
The ACK confirmation information of the package. In Ethernet, the MTU (Maximum Transmission Unit) value of TCP is 1460 bytes.
2. If the TCP_NODELAY option is set, the Nagle algorithm will be disabled, and the application will immediately call the data packet sent by Send.
Delivery to the network without delay.
To optimize performance at the application layer, Winsock copies the data sent by the application calling Send from the application buffer to Winsock.
Kernel buffer. Microsoft TCP stack uses a method similar to the Nagle algorithm to determine when data is actually delivered to the network.
The default size of the kernel buffer is 8 K. You can use the SO_SNDBUF option to change the size of the Winsock kernel buffer. If necessary,
Winsock can buffer data larger than the SO_SNDBUF buffer size. In most cases, the application completes the Send call, which only indicates data.
Copied to the Winsock kernel buffer, which does not indicate that the data is actually delivered to the network. The only exception is:
Disable the Winsock kernel buffer by setting SO_SNDBUT to 0.

Winsock uses the following rules to indicate to the application the completion of a Send call:
1. If the socket is still within the SO_SNDBUF quota, Winsock copies the data to be sent by the application to the kernel buffer to complete the Send call.
2. If the Socket exceeds the SO_SNDBUF limit and only one buffered data is sent in the kernel buffer, Winsock replication will send
To the kernel buffer to complete the Send call.
3. If the Socket exceeds the SO_SNDBUF limit and the kernel buffer has more than one buffered sending data, Winsock copies the data to be sent.
To the kernel buffer, and then ship the data to the network until the Socket falls to the SO_SNDBUF limit or only one data to be sent is left.
Complete the Send call.

Case 1
A Winsock TCP client needs to send 10000 records to the Winsock TCP server and save them to the database. Record size from 20 bytes to 100
Bytes. For simple application logic, the possible design scheme is as follows:
1. The client sends messages in blocking mode, and the server receives messages in blocking mode.
2. Set SO_SNDBUF to 0 on the client, disable the Nagle algorithm, and send each data packet separately.
3. The server calls Recv in a loop to receive data packets. Pass a 200-byte buffer to the Recv so that each record can be in a Recv call.
Obtained.

Performance:
During the test, it was found that the client could only send 5 pieces of data to the service segment per second, with a total of 10000 records, about KB. It took more than half an hour
To the server.

Analysis:
Because the client does not set the TCP_NODELAY option, the Nagle algorithm forces the TCP stack to wait for ACK confirmation of the previous data packet before sending the data packet.
Information. However, the client sets SO_SNDBUF to 0 and the kernel buffer is disabled. Therefore, only one data packet can be called for 10000 Send requests.
Packet sending and confirmation, each ACK confirmation message is delayed by 200 ms for the following reasons:
1. When the server obtains a data packet, it starts a 200 millisecond timer.
2. The server does not need to send any data to the client. Therefore, the ACK confirmation information cannot be carried along with the sent data packets.
3. The client cannot send data packets before receiving the confirmation information of the previous data packet.
4. When the timer on the server times out, ACK confirmation information is sent to the client.

How to improve performance:
There are two problems in this design. First, there is a latency problem. The client needs to be able to send two data packets to the server within 200 milliseconds.
Because the client uses the Nagle algorithm by default, the default kernel buffer should be used, and SO_SNDBUF should not be set to 0. Once TCP
The encapsulated data packet exceeds the MTU value. This data packet is immediately sent without waiting for the previous ACK to confirm. Second, this design
The scheme calls Send once for every small packet. It is not efficient to send such a small data packet. In this case
Add each record to 100 bytes and Send 80 records each time by calling Send. To let the server know the total number of records sent at a time,
The client can include a header before the record.

Case 2:
A Winsock TCP client opens two connections and communicates with a Winsock TCP server that provides the stock quotation service. First connection
The command channel is used to transmit the stock number to the server. The second connection is used as a data channel to receive stock quotations. After the two connections are established,
The client sends the stock number to the server through the command channel, and then waits for the returned stock quotation information on the data channel. The client receives the first
And then send the next stock number request to the server. Neither the client nor the server has set SO_SNDBUF and TCP_NODELAY.
.

Performance:
During the test, the client can obtain only five quotation records per second.

Analysis:

This design scheme allows only one stock information at a time. The first stock number is sent to the server through the command channel and received immediately
The stock quotation information returned by the server through the data channel. Then, the client immediately sends the second request message, and the send call returns immediately,
The sent data is copied to the kernel buffer. However, the TCP stack cannot immediately deliver this packet to the network because it does not receive the previous Packet
ACK confirmation information. 200 milliseconds later, the timer on the server times out, And the ACK confirmation information of the first request packet is sent back to the client.
To the network. The quotation information of the second request is immediately returned from the data channel to the client.
The timer has timed out, And the ACK confirmation information of the first quote information has been sent to the server. This process occurs cyclically.

How to improve performance:
Here, the design of two connections is unnecessary. If a connection is used to request and receive quotation information, the ACK confirmation information of the stock request will
The returned quotation information will be carried back immediately. To further improve performance, the client should call Send to Send multiple stock requests at a time.
Multiple quote information is returned at a time. If two one-way connections are required for some special reasons, TCP_NODELAY should be set for both the client and server.
Option, so that small data packets are sent immediately without waiting for the ACK confirmation information of the previous data packet.

Suggestions for improving performance:
The above two cases illustrate some of the worst cases. When designing a solution to send and receive a large number of small data packets, we should follow the following suggestions:
1. If data fragments do not need to be transmitted urgently, the application should splice them into larger data blocks and then call Send. Because the sending Buffer
It is very likely to be copied to the kernel buffer, so the buffer should not be too large, usually a little smaller than 8 K is very efficient. As long as the Winsock kernel buffer
To obtain a data block greater than the MTU value, several data packets are sent, leaving the last data packet. Except the last packet, the sender does not
Triggered by a timer of 200 milliseconds.
2. If possible, avoid one-way Socket data streams.
3. Do not set SO_SNDBUF to 0 unless you want to ensure that data packets are immediately delivered to the network after sending is called. In fact, 8 K buffer is suitable for most
The condition does not need to be changed again unless the new buffer is tested to be more efficient than the default size.
4. If data transmission does not require reliability, use UDP.

This article from the CSDN blog, reproduced please indicate the source: http://blog.csdn.net/bestdog/archive/2010/04/23/5516589.aspx