IP sharding technology Overview (from zkheartboy's Linux life)

In the Application Program The length of IP data must be concerned. If it exceeds the MTU of the network, the IP datagram must be split. If necessary, each network between the source and target ends must be split, not only when the sending host connects to the first network. In this article, we will discuss some principles of IP fragment and several elements required in the fragment (this article only discusses the maximum transmission unit of MTU and the maximum data segment of MSs ), in this paper, the Ethernet example is briefly decomposed. Key words: IP sharding, MTU, Mss introduction sharding is the embodiment of group exchange ideas and one of the two main problems solved by IP protocol. Slice in IP protocol Algorithm It mainly solves the differences between the maximum transmission unit (MTU) of the heterogeneous network, but the continuous partitioning and restructuring of the Group during the transmission process will bring a lot of work and increase some insecure factors. Body: 1. What is IP sharding? ip sharding is a technical means for transmitting IP packets on the network. When transmitting data packets, the IP Protocol divides the data packets into several parts for transmission and reorganizes them in the target system. This process is called fragmentation ). 2. Why does an IP shard usually occur when the size of the IP packet to be transmitted exceeds the maximum transmission unit MTU (maximum transmission unit. IP fragmentation usually occurs in the network environment. For example, in an Ethernet environment, the maximum IP packet size (MTU) can be transmitted is 1500 bytes. The size of the transmitted message is larger than 1500 bytes. In this case, you need to use the sharding technology to transmit the message only after it is split. In addition, using UDP can easily cause IP sharding, and it is difficult to force TCP to send a packet that requires sharding. 3. The principle of IP fragmentation and the process of analyzing fragmentation and re-assembly are transparent to the transport layer. The reason is that after the IP datagram is split, only when it reaches the next stop, can be re-assembled, and it is completed by the IP layer of the target end. The data packet after the sharding can also be sharded again as needed. The IP segment and the complete IP packet have almost the same IP header, and the ID domain is consistent for each segment. In this way, the parts of the same IP packet can be identified during re-assembly. In the IP header, the 16-bit identification number uniquely records the ID of an IP packet. IP segments with the same ID are reassembled; the 13-bit slice offset records the position of an IP slice relative to the entire package. The three-bit mark in the middle of the two tables indicates whether there are new slice after the slice. The three tags constitute all the information of the IP Shard, and the receiver can use the information to re-organize the IP data. 1. The role of a flag field. A flag field plays a major role in a fragmented datagram. When a data packet is sliced, its value is copied to each piece. One of the bits in the flag field is called the "not sharding" bit, and one of the bits is used to represent "more slices ". Except for the last piece, the bit must be set to 1 for each piece of data. The offset field refers to the position at the beginning of the offset of the original datagram. In addition, after the data packet is split, the total length value of each piece should be changed to the length value of the piece. If the bit of the flag field is set to 1, the IP address will not partition the datagram. Instead, discard the datagram and send an I c m p error message to notify the source host of the reason for rejection. If this is not a special requirement, 1 should not be set; the rightmost bit is set to 1, indicating that the message is not the last IP segment. If you intentionally send partial IP fragments instead of all, the target host will always wait for the fragments to consume and occupy system resources. This is the principle of some slice storm attacks. The following uses Ethernet as an example. Due to the electrical limitations of Ethernet transmission, each Ethernet frame has a minimum size of 64 bytes and a maximum size of 1518 bytes, drop the frame header of an Ethernet frame (DMAc destination MAC address 48bit = 6 bytes + SMAC source MAC address 48bit = 6 bytes + type domain 2 bytes) the CRC check for 14bytes and the end of the frame is 4 bytes, and the rest of the lines that carry the upper-layer protocol, that is, the data domain can only have 1500 bytes at the maximum, which is the MTU value mentioned above. This is also a concern of the network layer protocol, because the network layer IP protocol determines whether to partition the data transmitted from the upper layer based on this value. It is like a box that cannot hold a big chunk of bread. We need to cut the bread into slices and put it in multiple boxes. 2. MTU principle when two remote PCs are interconnected, their data must pass through many routers and a variety of network media to reach the peer end, MTU varies with different media in the network, which is equivalent to a long water pipe, which is composed of different thickness water pipes (MTU is different) the maximum water volume of a pipe is determined by the shortest pipe in the middle. For the upper-layer protocols at the network layer (the TCP/IP protocol family is used as an example here), they do not care about the pipe width. They think this is the network layer. The Network Layer IP protocol checks the size of each packet from the upper layer protocol, and determines whether to perform "sharding" based on the MTU size of the local machine. The biggest disadvantage of sharding is that it reduces the transmission performance. The tasks that can be done once are divided into multiple tasks. Therefore, a higher layer (that is, the transport layer) at the network layer is achieved) you will pay attention to this! For some reason, some high-rise personnel may require that I cannot slice this bread. I want a full bread, so I will add a label in the IP packet header: DF (donot fragment ). In this way, when the IP packet is transmitted in a large network (in the pipe), if the MTU is smaller than the IP packet, the forwarding device will discard the packet as required. Then, an error message is returned to the sender. This will often cause some communication problems, but fortunately most network links use 1500 or more MTUS. For UDP, this protocol itself is a connectionless protocol, which does not concern much about the arrival sequence of data packets and whether the packets arrive correctly. Therefore, UDP applications generally have no special requirements on fragment. The TCP protocol is different. This protocol is a connection-oriented protocol. For TCP, it is very concerned about the arrival sequence of data packets and whether errors occur during transmission. Therefore, some TCP applications have requirements on fragment-they cannot fragment (DF ). 3. MSS principle MSS is the maximum data segment that can be transmitted each time by a TCP packet. To achieve optimal Transmission Performance, TCP usually needs to negotiate the MSS value of both parties when establishing a connection, this value is often replaced by the MTU value when the TCP protocol is implemented (the size of the IP packet header must be reduced by 20 bytes and the packet header of the TCP data segment is 20 bytes). Therefore, the MSS is usually 1460. Both parties will determine the maximum MSS value for this connection based on the MSS value provided by both parties. When an IP datagram is sharded, each segment becomes a group with its own IP header and is independent from other groups when routing is selected. In this way, the data packets may be out of order when they arrive at the target end, but there is enough information in the IP header to allow the receiving end to correctly assemble the data packets. Although the IP sharding process Looks transparent, one thing you don't want to use is to re-transmit the entire datagram even if only one piece of data is lost. Because the IP layer itself does not have a mechanism for timeout retransmission-the higher layer is responsible for timeout and retransmission (t c p has a timeout and retransmission mechanism, but UDP does not. Some UDP applications also execute timeout and retransmission ). When a piece of data from the t c p packet segment is lost, t c p resends the entire t c p packet segment after the timeout. The packet segment corresponds to an IP datagram. There is no way to re-transmit only one piece of data. In fact, if the data packet is partitioned by an intermediate router instead of the starting system, the starting system cannot know how the data packet is partitioned. For this reason, we often need to avoid sharding. 5. Principles of the IP sharding algorithm fragment reorganization is one of the most important tasks of the IP layer. The main idea of its processing is: when a packet enters another Network B from one network, if the length of the data packet in the original network is greater than that of the maximum data packet in the other network, the data packet must be sliced. Therefore, there are several identification fields in the IP packet header that indicate the common identification number, the shard offset, whether the last shard and whether to allow the shard. The Gateway in the transmission uses these identification domains for sharding, and the host re-assembles the received parts to restore the data. Therefore, when a packet passes through network monitoring devices, security devices, and system management devices, data packets must be sliced or reorganized to obtain information and process data. Identification r df mf Fragment Offset R: Reserved unused; df: Don't fragment, "Not sharding" bit. If this bit is set to 1, the IP layer will not partition the datagram; mf: More fragment, "more slices". In addition to the last segment, BITs must be set to 1 for each part of the datagram; Fragment offset: the starting position of the original packet offset. The offset value is multiplied by 8 bytes. 6. Summary This paper describes what IP sharding is and illustrates the current image of IP sharding and its features. This article briefly discusses the principles of IP sharding. MTU is analyzed in detail, the role of MTU in IP Fragment technology is described, ICMP inaccessibility error is briefly discussed, and MTU discovery process is observed using UDP. This article briefly describes the role of MSs and how the value is obtained. This article only makes a small discussion about IP sharding.