In-depth UDT protocol analysis

Over the past two years, we have chosen the UDT protocol for data transmission because we wanted to choose a protocol that combines the reliability of the TCP protocol and the efficiency of the UDP protocol. We have also studied and practiced it for a while. However, in the end, it was abandoned because it was not suitable for our business scenarios. However, I was deeply impressed by its implementation principle, code implementation, and excellent application performance, today, I took a summary note to review the UDT protocol.

• What is the UDT protocol? It is a UDP-based data transmission protocol (UDT ).

• What is the main role of the UDT protocol? The main purpose of UDT is to support massive data transmission on high-speed wide-area networks, while the standard data transmission protocol TCP on the Internet has poor performance in high-bandwidth Long-Distance Networks.

• What is the difference between UDT and UDP? Built on UDP, UDT introduces new congestion control and data reliability control mechanisms. UDT is a two-way connection-oriented application layer protocol. It also supports reliable data stream transmission and partial reliable data packet transmission.

• What are the application scenarios of UDT? Because UDT is fully implemented on UDP, it can also be applied in other application fields besides high-speed data transmission, such as point-to-point technology (P2P), firewall penetration, and multimedia data transmission.

(The answer to the above questions is taken from Wikipedia.) Of course, today I am not a knowledge Porter, but I will analyze it in the UDT protocol based on the basic definition of the above UDT protocol.

What are the main features of the UDT protocol?

• UDP-based application layer protocol: Anyone with basic network knowledge knows the Differences Between TCP and UDP and Their Use Cases, however, if there is one protocol that can take into account both the security and reliability of TCP and the efficiency of UDP, then UDT is one.

• Connection-oriented protocol: connection-oriented means that two applications using the Protocol must establish a connection before exchanging data with each other. Of course, UDT is a logical connection channel. The maintenance of such connections is based on handshaking, keep-alive (keep alive), and closing the connection.

• Reliable protocols: the packet sequence number mechanism, the receiver's ack response and packet loss report, the ACK sequence number mechanism, and the retransmission mechanism (based on the packet loss report and timeout processing) are used to ensure the reliability of data transmission.

• Duplex Protocol: Each UDT instance contains information about the sender and receiver.

• Unicast data streams.

• The new congestion control algorithm has a scalable congestion control framework: the new congestion control algorithm is different from the window-based TCP congestion control algorithm (slow start and congestion avoidance ), it is a hybrid window-based, rate-based congestion control algorithm. Scalable congestion control framework open-source code and congestion control C ++ class architecture, supports developers to derive dedicated congestion control algorithms.

• Bandwidth Estimation: UDT uses the packet (PP-Packet Pair) mechanism to estimate the bandwidth value. That is, each 16 packets is a group, and the last one is a pair of packets, that is, the sender does not have to wait until the next sending cycle before sending. After receiving a packet, the receiver records its arrival time. The bandwidth of the link can be calculated based on the value of the previous record (the calculation method is called the median filter method ), and give feedback in the next ack.

Implementation of some main features of UDT

The UDT packet validation mechanism is implemented based on time timers. Principle:

Uses timer-based selective acknowledgement, which generates an acknowledgement at a fixed interval. this means that the faster the transfer speed, the smaller the ratio of bandwidth consumed by control traffic. meanwhile, at very low bandwidth, UDT acts like protocols using cumulative acknowledgement.

The ack interval of UDT is the same as the rate control interval (SYN ).

To support this scheme, nega tive acknowledgement (NAK) is used to explicitly feed back packet loss. nak is generated once a loss is detected so that the sender can react to congestion as quickly as possible. the loss information (sequence numbers of lost packets) will be resent after an increasing interval if there are timeouts indicating that the RETR ansmission or Nak itself has been lost.

UDT traffic control is implemented based on the following three mechanisms.

• New Congestion Control

• Daimd Rate Control

• Dynamic window control

Detailed description in this: http://www.jenkinssoftware.com/raknet/manual/congestioncontrol.html

Data transmission types supported by UDT

• Stream-based send and Recv.

• Based on the datagram sendmsg and recvmsg.

• File transmission sendfile, recvfile.

Next we will use the UDT version 4 to analyze some new features of this version of UDT.

• The UDP multiplexer (UDP multiplexing) mechanism is used. The advantages of this mechanism are:

Therefore it is possible (and by default) All UDT sockets in one process will share one UDP port. This scheme makes it easier for firewall traversing.

• UDT traffic control adopts a configurable congestion control algorithm. You can disable it, configure it, and improve it to implement the traffic control policies you need.

• Use new resource management (memory management) and shared congestion control methods to support more concurrent UDT connections. Memory Management is described as follows:

  • Udt4 has a new buffer management module that enables all UDT sockets in one process can share protocol buffer. The goodness it brings is the much less memory usage for multiple UDT connections compared to previous versions.

  • Udt4 can automatically resize its buffer in order to reduce memory usage while providing maximum throughput.

  • Because of the new memory management scheme, overlapped Io has been removed from udt4. if your existing code uses overlapped Io, You need to modify it to use regular Io. this is the only change needed for exiting code to move from udt3 to udt4.

• Other features:

  • Does not conflict with native Socket API;

  • Thread security;

  • Handles are not shared between processes;

  • Handle errors;

  • Firewall penetration;

  • High Security;

  • Fast connection Establishment;

The above is my analysis and summary of the UDT protocol based on my own memories and notes. Here, I will study the UDT protocol. MMAP is welcome to download it. We are very grateful to Dr. Gu yunhong for providing us with such an outstanding Agreement.