Analysis of the return time of the RECV function of TCP protocol socket in Linux (sticky packet)

Http://www.vckbase.com/index.php/wv/10
http://blog.csdn.net/zlzlei/article/details/7689409

Article One:

Currently in the network transmission application, the TCP/IP communication protocol and its standard socket application Development Programming Interface (API) are widely adopted. The TCP/IP transport layer has two parallel protocols: TCP and UDP. TCP (Transport Control Protocol, transmission Protocol) is connection-oriented and provides high reliability services. UDP (User Datagram Protocol, Subscriber Datagram Protocol) is non-connected and provides efficient service. In practical engineering applications, the choice of reliability and efficiency depends on the environment and requirements of the application. In general, the network transmission of ordinary data uses high-efficiency UDP, and the network transmission of important data uses TCP with higher reliability.

In the application development process, I found that applications based on TCP network transmission sometimes appear sticky packet phenomenon (that is, the sender sends a number of packets of data to the receiving party to stick to a packet). In view of this situation, we have carried on the special research and the experiment. In this paper, the problem of TCP network sticky packet is analyzed emphatically, and the countermeasures and methods for solving this problem are put forward, which are used for reference by engineering technicians.

I. Introduction to the TCP protocol

TCP is a connection-oriented transport layer protocol, although TCP does not belong to the ISO set of protocols, but because of its successful application in the business and industry, it has become a de facto network standard, widely used in various network host communication.

As a connection-oriented transport layer protocol, the goal of TCP is to provide users with reliable end-to-end connections to ensure the orderly and accurate transmission of information. In addition to providing basic data transmission functions, it also adopts a series of measures such as data number, checksum calculation and data confirmation to ensure reliability. It numbers each byte of data that is transmitted and requests the receiver to return a confirmation message (ACK). If the sender does not receive the data confirmation within the specified time, it will retransmit the data. The data number allows the receiver to handle the disorder and repetition of the data. The data error problem is solved by adding a checksum to each transmitted data segment, and the receiver checks the checksum after receiving the data, and if the checksum is incorrect, the data segment of the error code is discarded and the sender is required to retransmit it. Flow control is also an important measure to ensure the reliability, if no flow control, may be due to receive buffer overflow and loss of large amounts of data, resulting in a lot of retransmission, causing network congestion vicious circle. TCP uses a variable window for flow control, and the receiver controls the amount of data sent by the sender.

TCP provides users with high-reliability network transmission services, but reliability assurance measures also affect the transmission efficiency. Therefore, in the actual engineering application, only the transmission of the key data is TCP, and the transmission of common data generally uses the high efficiency UDP.

Analysis on the problem of sticky bag and its countermeasures

A TCP sticky packet is a packet of packets sent by the sender to the receiver when it is received, viewed from the receive buffer, followed by the head of the packet data immediately preceding the end of the packet.

There are many reasons for the sticky-packet phenomenon, which may be caused by the sender or by the receiver. The sticky packets caused by the sender are caused by the TCP protocol itself, and TCP is often needed to collect enough data to send a packet of data to improve transmission efficiency. If the data sent several times in a row is very small, usually TCP will be based on the optimization algorithm to synthesize the data packets sent out once, so that the receiver received the sticky packet data. The sticky packet caused by the receiver is due to the fact that the receiver user process does not receive the data in time, resulting in sticky packets. This is because the receiver first put the received data in the system receive buffer, the user process from the buffer to fetch data, if the next packet of data arrives before a packet of data has not been taken away by the user process, the next packet of data into the system receive buffer when the previous packet of data is received, The user process takes the data from the system receive buffer based on the pre-set buffer size, so that it takes more than one packet of data at a time (as shown in Figure 1).

Figure 1

Figure 2


Figure 3

There are two kinds of sticky packets, one of which is glued together is the complete packet (Figure 1, Figure 2), the other case is stuck together with the package has an incomplete package (Figure 3), this assumes that the user receive buffer length of M bytes.

Not all sticky-packet phenomena need to be processed, if the transmitted data is a continuous stream of data without structure (such as file transfer), you do not have to separate the packets of adhesion (short-subcontracting). But in the actual engineering application, the transmitted data is usually the structure data, then needs to do the subcontract processing.

When dealing with the sticky-packet problem of fixed-length structure data, the sub-packet algorithm is simple, and the sub-packet algorithm is more complicated when it deals with the sticky-packet problem of uncertain long structure data. Especially in the case of 3, as a packet of data content is divided into two successive receiving packets, the processing is more difficult. In practical engineering application, the phenomenon of sticking should be avoided as far as possible.

In order to avoid sticking, the following measures can be taken. One is caused by the sender of the sticky packet phenomenon, the user can be programmed to avoid, TCP provides a mandatory data transfer immediately after the operation of the instructions PUSH,TCP software received the operation instruction, the data immediately sent out, without waiting for the transmission buffer full, and the receiver caused by the sticky packet, By optimizing the program design, reducing the workload of receiving process, improve the priority of receiving process and so on, so that it can receive data in a timely manner, so as to avoid the phenomenon of sticky packet; the third is controlled by the receiver, a packet of data by the structure of the field, the human control sub-multiple reception, and then merge, through this means to avoid

The above mentioned three kinds of measures, all have their shortcomings. Although the first method of programming can avoid the sticky packets caused by the sender, it shuts down the optimization algorithm, reduces the network sending efficiency, affects the performance of the application, and is generally not recommended for use. The second method can only reduce the likelihood of sticky packets, but does not completely avoid the sticky package, when the sending frequency is high, or because the network burst may make a time period packet arrives at the receiver faster, the receiver may still be too late to receive, resulting in sticky packets. The third approach avoids sticky packets, but the application is less efficient and unsuitable for real-time applications.

One of the most comprehensive countermeasures is that the receiver creates a preprocessing thread, and the received packets are preprocessed to separate the stuck packets. We have experimented with this method and proved to be efficient and feasible.

Third , programming and implementation

1. Implementation Framework

The Experiment Network communication program uses the TCP/IP protocol Socket API programming implementation. The socket is for the client/server model. The TCP implementation is shown in Framework 4.

Figure 4

2. Experimental hardware environment:

Server: Pentium 350 microcomputer

Client: Pentium 166 microcomputer

Network platform: A LAN connected by a 10-Gigabit shared hub

3. Lab Software Environment:

Operating system: Windows 98

Programming language: Visual C + + 5.0

4. Main thread

Programming in a multi-threaded way, the server has a total of two threads: Send data thread, send statistics display thread. There are three threads in the client: receiving a data thread, receiving a preprocessing sticky packet thread, receiving a statistics display thread. Where the send and receive thread priority is set to thread_priority_time_critical (highest priority), the preprocessing thread priority is thread_priority_above_normal (higher than the normal priority), The display thread priority is thread_priority_normal (normal priority).

The data structure that the experiment sends is shown in 5:

Figure 5

5. Sub-packet algorithm

According to the three kinds of different sticky-packet phenomena, the sub-package algorithm adopts the corresponding solution respectively. The basic idea is to first convert the received data stream (length set to m) to a predetermined structure data form, and remove the structure data length field, which is the N in Figure 5, and then the first packet data length according to n calculation.

1) If n

2) If n=m, it indicates that the data stream content is exactly a complete structure data, directly into the temporary buffer.

3) If the n>m, it indicates that the data stream content is not enough to form a complete structure of data, to be left with the next packet of data to be processed.

Interested in the specific content of the subcontracting algorithm and the software implementation, can contact the author.

Iv. Analysis of experimental results

The experimental results are as follows:

1. In the above experimental environment, when the sender of several packets of packet length of the sum of less than 1500b, often appear sticky packet phenomenon, the receiver after preprocessing thread processing can correctly solve the sticky package. If send no delay is set in the program: (SetSockOpt (Socket_name,ipproto_tcp,tcp_nodelay, (char *) &on,sizeof on), where on=1), there is no sticky-packet phenomenon.

2. When sending data to the 1kb~2kb of each packet, if the sending interval is less than 10ms, occasionally sticky packets will appear, and the receiving party can correctly unpack the packets that stick together after the preprocessing thread processing.

3. In order to determine the time of processing the adhesive packet, the sending party sequentially sends the length 1.5kb, 1.9kb, 1.2kb, 1.6kb, 1.0kb data, a total of 1000 packets. In order to make the sticky packet phenomenon, the receiving thread waits for 10ms before each receive, the receive buffer is set to 5000b, and the receiving party receives 526 packets of data, of which 175 packets are in length 5000b. The preprocessing thread processing can get 1000 packets of correct data, the total time of the adhesive packet processing is less than 1ms.

The experimental results show that the TCP sticky packet phenomenon exists, but it can be solved by the preprocessing of the receiver, and the processing time is very short (the 1000 packet data processing time is less than 1ms in the experiment), which hardly affects the normal operation of the application.

Article Two:

Used to find people on the internet before the recv when the return, or said is very general, or completely feel not reliable, recently or to do an experiment to analyze it:

Test 1.
Size per send: 1024
Each receive size: 32
Results: Pack1
Send a packet per send, packet size 1024, with push flag
The RECV function returns after each receive full 32.


Test 2.
Size per send: 1024
Each receive size: 2048
Results: Pack2
Send a packet per send, packet size 1024, with push flag
The RECV function returns after each receive full 1024.


Test 3.
Size per send: 20480
Each receive size: 10240
Results: Pack3
Each send sends two packets with a packet size of 16384 (TCP shard size, agreed upon establishing connection) with 4096, only the last packet with the push flag
The RECV function returns after the first time the full 10240 is received
recv function return after second receive 6144 (6144+10240=16384)
After the third receive 4096, the RECV function returns
Recv >>> [1]10240:10240
Recv >>> [2]10240:6144
Recv >>> [3]10240:4096
Recv >>> [4]10240:10240
Recv >>> [5]10240:6144
Recv >>> [6]10240:4096




Test 4.
Size per send: 20480
Each receive size: 20480
Results: Pack4
Each send sends two packets with a packet size of 16384 (TCP shard size, agreed upon establishing connection) with 4096, only the last packet with the push flag
The RECV function returns after receiving 16384 for the first time
After the second receive 4096, the RECV function returns
Recv >>> [1]20480:16384
Recv >>> [2]20480:4096
Recv >>> [3]20480:16384
Recv >>> [4]20480:4096


Test 5.
Size per send: 400000
Each receive size: 10240
Results: PACK5
In addition to the last packet sent not unexpectedly, the data are sent in accordance with the packet size of 16384, the process of sending regular receive window full.
The receiving process is received at the initial 16384 packet size, with no regularity at the late stage:
Recv >>> [1]10240:10240
Recv >>> [2]10240:6144//a//full 16384
Recv >>> [3]10240:10240
Recv >>> [4]10240:6144//b//full 16384, from start to full 32768 (sliding window total size)
Recv >>> [5]10240:10240
Recv >>> [6]10240:10240
Recv >>> [7]10240:10240
Recv >>> [8]10240:2048//c//from start to here full 65536 (two sliding window total size)
Recv >>> [9]10240:10240
Recv >>> [10]10240:6144//d//full 16384
Recv >>> [11]10240:10240
Recv >>> [12]10240:6144//e//full 16384
Recv >>> [13]10240:10240//f//will be able to fill the buffer
Recv >>> [14]10240:10240
。。。。。。 It's all 10,240:10,240.
Recv >>> [40]10240:10240
Recv >>> [41]10240:8192//e//
Recv >>> [42]10240:6784


Test 6.
Size per send: 400000
Each receive size: 10000
Results: Pack6
In addition to the last packet sent, the data is sent in accordance with the packet size of 16384, the regular receiving window is full during the sending process.
The receiving process is received at the initial 16384 packet size, with no regularity at the late stage:
Recv >>> [1]10000:10000
Recv >>> [2]10000:6384
Recv >>> [3]10000:10000
Recv >>> [4]10000:6384
Recv >>> [5]10000:10000
Recv >>> [6]10000:10000
Recv >>> [7]10000:10000
Recv >>> [8]10000:2768
Recv >>> [9]10000:10000
Recv >>> [10]10000:6384
Recv >>> [11]10000:10000
Recv >>> [12]10000:6384
Recv >>> [13]10000:10000
。。。。。。 It's all 10,000:10,000.
Recv >>> [41]10000:10000
Recv >>> [42]10000:4912
Recv >>> [43]10000:6784


Test 7.
Each send size: 32, Hair 10,000 times
Each receive size: 102400
Results: Pack7
Most of the packages are sent in 32 size, but some packages are sent together, and each packet sent with a push flag
The majority of the packets received during the receive process are received at 32 size, but there is a time when the receive is greater than 32, and there is no relationship with sending packets larger than 32.
Recv >>> [1]102400:32
Recv >>> [2]102400:32
Recv >>> [3]102400:32
。。。。。。 It's all 102,400:32.
Recv >>> [92]102400:32
Recv >>> [93]102400:49184
Recv >>> [94]102400:32
。。。。。。 It's all 102,400:32.
Recv >>> [151]102400:32
Recv >>> [152]102400:32
Recv >>> [153]102400:22720
Recv >>> [154]102400:32
。。。。。。 It's all 102,400:32.
Recv >>> [268]102400:32
Recv >>> [271]102400:47840
Recv >>> [272]102400:32
。。。。。。 It's all 102,400:32.
Recv >>> [397]102400:32
Recv >>> [398]102400:32
Recv >>> [399]102400:59776
Recv >>> [400]102400:32
。。。。。。 It's all 102,400:32.
Recv >>> [523]102400:32
Recv >>> [526]102400:54208
Recv >>> [527]102400:32
。。。。。。 It's all 102,400:32.
Recv >>> [630]102400:32
Recv >>> [631]102400:32
Recv >>> [632]102400:65568
。。。。。。 It's all 102,400:32.
Recv >>> [652]102400:32
Recv >>> [653]102400:32


Analysis: The user reads from the receiving buffer and the kernel to the receiving buffer to fill the content of the two processes are mutually exclusive,
Recv only gets the content once from the receiving buffer, and only cares about how much content it receives when it acquires the buffer:
A, if there is no data in the buffer, then recv into the blocking state, waiting for the kernel buffer is filled into the data after reactivation.
b, if the data in the buffer is larger than the buffer used by the user, the RECV function returns after filling the buffer.
C, if the data in the buffer is smaller than the buffer used by the user, the RECV function is returned after taking all the data in the buffer.
Kernel to the receiving buffer is to receive the packet to its own time slice and the buffer is not read, then will get the package content into the buffer.


This allows you to interpret the phenomena in Test 5:
A, kernel the 12th time only received a packet of 16384 size, so received A and B have received the entire packet
b, slowly the arrival of a succession of land kernel the third time received two 16384-size package, filled the entire buffer, so received C,
In the user's time on-chip users collected all the data in the buffer.
C, the next data in droves, causing the buffer has been in full state, so the subsequent reception can be filled with user buffer.
D, receive the E at the end of the entire buffer, and then receive the closure of the package
To verify the above inference, test 6 was performed.


Test 7 verifies the sending and receiving situation at the time of the fast send, proves that send sends and recv receive process completely does not matter,
Even if a packet is sent with a push tag, it can only guarantee that each packet is placed in the receive buffer separately, and that the RECV receive a packet return per receipt.
The entire receiving process follows the rules above, when to return, and how much data to return depends on how much data is in the buffer.
Like most packages in test 7, each package is returned just because the receive is faster, causing kernel to put a packet in the receiving buffer at a time.


Conclusion:
A, do not expect the use of TCP protocol one party send once, the other party recv logic, this logic is unreliable, there is no theoretical basis
B, in the local control of the message logic, or signaling level transmission, it is recommended to use UDP
C, if you have to use TCP message transmission must be added synchronization header, such as a fixed value, used to split the packet or verify whether the packet is disorderly or out of bounds

Analysis of the return time of the RECV function of TCP protocol socket in Linux (sticky packet)