On the DHT method in BitTorrent protocol

background: BT protocol

1. Ideas and causes of peer
Peer to peer. The first thing to understand about peer is a more primitive structure: c/S structure.

In C/s structure, client,s refers to the server. This is the most common kind of structure, such as Taobao shopping, Ali has a dedicated server centralized for the country's users to provide services. All the work is done by a central server.

The C/s structure has his advantages, but in some cases it may not be very effective. For example, if you download a movie, if someone shares a movie from their computer, then if all the other users on the network are connected to the share's computer for download, the share will not be able to withstand such a large amount of traffic. Then how to solve it. Would you like to rent a distributed server cluster dedicated to sharing videos. Although it is not impossible, but this is bound to increase the difficulty of file sharing. As a result, the peer method appears.

Peer to peer method is the only user, a client. In this structure, users can transfer data to users, not all users from the same server to obtain data. By sharing data between users to reduce the pressure on the server, this is the idea of the peer-to approach.

2, BitTorrent Agreement Foundation
BT protocol has undergone several generations of change, here we first talk about the most primitive BT protocol.

Be aware that the BT protocol is a typical peer application. If you want to implement the download between users, the most direct question is how to know the users who download the same resource.

This is the work of the tracker server. Tracker as the name implies, the job is to track download user information. A seed that corresponds to an active user list, and when a new user wants to download the seed, it returns the active user list to the new user and adds the user to the list so that the new user can download the data from other users.

With the tracker server, you can know the information of other users. Then there is a missing link, that is how to get tracker server information. The answer is from the seed file. The seed file contains the tracker server address for the seed file.

In addition to the address of the tracker server in the seed file, there are also checksum values for each block of the file. When the file is shared by the BT protocol, it is divided into several file blocks, and the exchange files between the users are in blocks. When the file is received, it will need to be re-downloaded than the corresponding checksum value in the seed file.

3. Defects and improvement methods of the original BT protocol
As can be seen, the original BT protocol is still a centralized structure, tracker server as the core of the entire system, once the failure, the corresponding seed files will be invalidated. De-centering is an improvement that the BT protocol is very much in need of. After the centralized BT protocol, should be able to go from the network to find their own needs of other users.

There were two improvements to the original BT protocol, and the first failed improvement was to look for the active user in a broadcast manner, and the user who received the broadcast forwarded the broadcast until an active user of the seed file was found. This method is very easy to trigger a "broadcast storm", so quickly abandoned.

Another way to improve this is to use the DHT method so far. Next, start with the DHT method. DHT Structure

DHT = Distributed hash table Chinese distributed hash table.
In a word, the role of DHT is to replace the tracker server and store the information stored in the tracker in a distributed way by storing it in a different user.

With DHT, the process of downloading the seed file becomes: First, query the active user of the seed from DHT. Then, start the peer download based on the user information obtained. What's different from the original BT protocol is that it changes the way you get active users.

Thus DHT can be said to be a distributed database. A DHT network consisting of all users in the network is a large database. Each user stores a small amount of data.

Understanding DHT requires a few important questions to understand:
1, how to allocate the storage location of data. That is, how to know which node the data should be stored on.
2, how to design the routing algorithm. Update of the routing table.
3, the data query, add.

The DHT method is described in detail below.
(to illustrate, DHT is a conceptual approach that is implemented in a number of ways, only the Kademlia method, which is the method used in the BT protocol.) ) how data is distributed in DHT

Data in DHT is stored in the form of key value pairs (key value pairs). In the application of BT protocol, key is the id,value of the seed file and is the active user (peers) information of the seed file. The seed ID here is the infohash of 160bit, which is the unique string generated by the hash of each seed file.

Each storage unit in DHT is, in fact, a user, called node, that is distinguished from peer. Each node also has an ID, which is a randomly generated 160bit string. So, the data ID and the Nodeid are all 160bit strings.

The next step is to introduce a concept to pave the way--distance. Between node and node, there is a distance between node and the data, and the distance is calculated by doing an XOR operation of two 160bit IDs. Note that the node ID is randomly generated, so the distance here is independent of the actual network connection.

The basic thought of data storage in DHT is to store the data in node closest to it, that is, to store the data in the node with the closest ID and data ID. To be exact, the closest few node. Routing in DHT

Knowing the data and node distance calculation and storage principles in DHT, we know that the target data can be found as long as the number of node closest to the data ID is located. One problem, however, is that in a node, you can store the ID of some other node and the corresponding IP address port number. However, you cannot store information for all nodes. To connect to the target node, you need to ask the other nodes and slowly approach the target nodes.

1. Buckets
The storage node ID, IP address, and port number list is called the routing table. To put it simply, the more node is closer to the node in the routing table, the farther away it becomes. Specifically, a structure called a bucket is used.

A bucket structure is a collection of information from other nodes in the routing table and can hold up to 8 node in a bucket. Node initially has only one bucket, and the range is the entire space of the 160bit ID. If a bucket is full and the bucket contains its own node ID, then the bucket is allowed to be split in the average. If the bucket range does not include its own nodeid, then it cannot be split. As can be seen, the continuous division, the new bucket range will be more and more small, and closer to the Nodeid. Since each bucket can only save 8 node, it has ensured that there is more node information near the distance, and less node information is far away.

2. Inquiry Process
The query process for data in DHT is that the node that initiates the query first calculates the distance from the node in the routing table to the target data, and then initiates a data query on the nearest K node. The node that receives the data query request, if there is data returned directly, does not query the closest node of the data in its own routing table and returns. The node that initiates the query receives the returned nodes, and then continues to query those nodes, knowing that the required data is returned.

This continuous query process steps closer to the node where the data is present.

3, the update of the routing table
Users in DHT are constantly changing, so some mechanisms are required to ensure that routing table information is not obsolete.

node in the routing table has three states: good, questionable, bad. Simply put, good is an active node. If there is no information on a node for a long time, the node becomes questionable, and if there is no response to a node sending requests multiple times, the node is bad and is removed from the routing table.

The questionable node is temporarily not removed, and the questionable node in the bucket is not tested until the new node is added but the bucket is full, and the nodes that are not responding are removed.

In addition, in each bucket, at least one good node needs to be maintained (so as to ensure that any point in the entire network is accessible), if all the points in a bucket are not static for a long time, you need to refresh the entire bucket, The Refresh method is to initiate a query within the bucket range.

4. New node initialization routing table
When a new node enters a DHT network, at least one node must be known. The query is then initiated against the node, and its own ID is queried. In this way, it will return more and more close to its own node information. The node information is stored during the query. The initialization of the routing table is completed after the query is finished. data Read-write and Data retention

1, increase and revise the search
The query process of the data is mentioned earlier. Modify the data method is similar, also first query data ID, finally can get a few node, modify the data on these node.

2. Coping with node changes
The nodes are constantly changing, and the K nodes that store the data may all be offline and result in data loss.

The refresh mechanism can deal with this problem, the refresh mechanism is simple, that is, every time, all the nodes have their own data to be stored again. is to re-query the data ID and store all of your data in the returned node. This method can not guarantee that the data is not lost, but the probability of loss is relatively low.

How does the new join node handle it. The new node initializes the routing table during the join process, and the node is discovered by other nodes in the process. The other node determines the distance between the new node and the data it stores, and stores the data on the new node if the new node is close enough. References

[1] BitTorrent Community document DHT Protocol
[2] (DHT methodology article) kademlia:a Peer-to-peer Information system based on the XOR metric
[3] (a good post) talk about the principle of distributed hash table (DHT)--taking Kademlia (Kad) and Chord as examples

PS: This article is intended to briefly describe DHT, omitting some of the details, and partly imprecise. For more information, please read resources.