Step-by-step tutorial to write a BT seed sniffer two---DHT

This is a creation in Article, where the information may have evolved or changed.

Before writing the principle, in the principle of a simple introduction of DHT, but not enough detail. Today we'll talk about the core of the sniffer-dht, which is the default principle you've read.

Background knowledge

DHT full name distributed hash table, Chinese translation comes from a distributed hash table. It is a decentralized distributed system, characterized by automatic de-centering, strong fault tolerance, support expansion. It also provides its own architecture, including the Keyspace and overlay network (coverage networks) two parts. However, he did not specify the details of the algorithm, so there are many different ways of implementation, such as Chord,pastry,kademlia. DHT in BitTorrent is based on a variant of Kademlia, whose official name is Mainline DHT.

The DHT person, as its name, sees it as a whole, from a distance, it is a hash table, except that it is distributed and exists on many machines. It also supports set (key, Val), get (key) operations. DHT can be used in many ways, such as Distributed file Systems, DNS, instant messaging (IM), and the most familiar point-to-point file sharing (such as BT Protocol).

The DHT we mentioned below is mainline DHT by default, and examples are represented by pseudo-code. when you read the following paragraph, always remember that DHT is a hash table .

Mainline DHT

Mainline DHT follows the structure of DHT, which we specify from both the Keyspace and overlay network respectively.

Keyspace

Keyspace is mainly about key rules.

Mainline DHT inside the key length is 160bit, note is bit, not byte. In the common compiled programming language, the longest integer type is also 64bit, so the integer type is not the key, we have to think of other ways. We can use arrays to represent it, array types you can use different lengths of the integer type, such as Int8,int16,int32. Here we use a byte array of length 20 for the convenience of the below calculation.

In mainline DHT, the only type of calculation between keys is XOR, or XOR (remember the different or the knowledge?). ）。 Our key is represented by a byte array of length 20, so we should calculate the XOR value of the corresponding byte of the two key from the go, and the end result is another byte array of length 20. The algorithm is as follows:

for i = 0; i < 20; i++ {    result[i] = key1[i] ^ key2[i];}

Read here, do you want to ask what is the use of XOR? Do you remember how DHT works in the theory?

XOR is to find the nearest K node in a friend's table, what is the closest node? Is that each node and key in a friend is different or, the smaller the result, the closer. Here another question is derived, how to compare the size between byte arrays? It is simple to compare the size of each byte from the point of travel.

In mainline DHT, we use the 160bit key to represent each node and the ID of each resource, and when we look for a node or find a resource, we actually look for their ID. Recall that this is not a hash table? :)

Another smart you might ask, how do we know the ID of each node or resource? In mainline DHT, the ID of a node is generally generated randomly, and the ID of the resource is obtained by encrypting the contents of the resource with the SHA1 algorithm.

OK, about the key is so much, code implementation you can examine here.

Overlay Network

Overlay network is mainly about how the DHT internal nodes store data, and how the different nodes communicate.

First, let's look back at how DHT works in this article:

DHT consists of a number of nodes, each of which holds a table that records its own friend node. When you query a node A for another node B's information, a will query its friend table, if it contains B, then a will return B information, otherwise a will return the distance B distance from the nearest K node. Then you re-query the K-node information of B, so that the loop until the query to B information, query to the information of B you should send a notification to all previously queried nodes, tell them that you have B information.

The entire DHT is a hash table that piecemeal its own data in different nodes. OK, now let's look at what data structures are stored inside a node.

Node internal data storage-Routing Table

What kind of data structure to use to see what kind of operation to support, but also depends on the frequency of various operations. From the way it works we know that there are two main operations:

• In my Friends node (note: "I" is a node) query the nearest K node from a key (in mainline DHT, k=8), the degree is frequent

• Save a node, that is, insert operation, the degree is frequent

First See "Recent", "K", we will associate the top K problem. A very straightforward approach is to save the node with an array. In this case, we look at the complexity of the algorithm. Top k problem is solved with heap, query complexity is O (k + (N-k) *log (k)), when K=8, close to O (n); insert operation is O (1). Note: n is the total number of buddy nodes for one node.

When n is very large, the operation time can be very long. So is there an O (log (n)) algorithm?

Lenovo to the above heap of algorithms, you may say, we can maintain a heap ah, insert and query the time is O (log (n)). The heap is maintained based on the distance of the elements in the heap from a fixed key, but often the key we query is variable, so this is not possible.

Is there another O (log (n)) algorithm?

Experience tells us that many O (log (n)) problems are related to two fork trees, such as various balanced binary trees, can we use a binary tree to solve it? Lenovo to each ID is a 160bit value, and we know that the distance between key is calculated by XOR, and the difference or result size of the two key is independent of their common prefix, we should think of trie tree (or called prefix tree) to solve. In fact, the trie tree is used in the Mainline DHT protocol, but slightly different from the trie tree. In the trie tree, when inserting a key, we compare each char and trie inside the key to the inner path, and when inconsistent, it splits into a subtree immediately. But here, when there is inconsistency, there is no immediate division, but a buffer of length k (called Bucket in mainline DHT). There are two kinds of situations to discuss:

• If the bucket length is less than k, insert the bucket directly.

• If the bucket length is greater than or equal to K, there are two different situations to discuss:

  • The first case is that the current path is the node ID (note that it is not the key to insert, the "I" own id) prefix, then split, the left and right sub-tree keys are 0 and 1, and the current bucket node according to their current char value into the corresponding subtree of the bucket inside.

  • The second case is that the current path is not a prefix for the node ID, in which case the key is discarded directly.

If you don't understand it, you can refer to the diagram above in Kademlia's paper.

When inserting, the complexity is O (log (n)). When querying the K-node nearest to key, we can find the bucket of the current key, if there is not enough k in the bucket, then we find the node precursor and successor, finally, according to their distance from the key to shoot the order, the average complexity is O (log (n)). This inserts and queries are O (log (n)).

Code implementation you can examine here.

Communication between nodes-KRPC

KRPC is relatively simple, it is a simple RPC structure, which transmits messages through UDP, the message is a dictionary encoded by Bencode. It contains 3 message types, request, response, and error. The request is divided into four kinds: Ping,find_node, get_peers, Announce_peer.

• Ping used to detective each other whether online

• Find_node is used to find specific information about a node ID key, including Ip,port,id

• Get_peers is used to find specific information about a resource ID key, which contains a list of ip:port to download the resource.

• Announce_peer is used to tell others that they can provide a download of a resource and let others save the message. Do you remember angelababy that example? After I get her number, I will inform all of the people I have asked before, they will be I have Angelababy number this information to save, and later if someone asks if they have Angelababy number, they will tell that person I have. The BT Seed sniffer is based on this to get the message, but we still need to download it after we get the message .

Jump out of the node, look at DHT as a whole hash table, Find_node and get_peers is what we said before the get (key), Announce_peer is set (Ke, Val).

The rest is the exact message format, which you can see in the official documentation, where you don't move bricks.

When implementing KRPC, the following points need to be noted:

• Each time you receive a request or reply you will need to update your routing Table as appropriate, or save or discard it.

• You need to implement transaction,transaction inside to include your request information and the requested IP and port, only so that when you receive a reply message, you can be based on the message's transaction ID to make the correct processing. Mainline DHT does not specify how to implement transaction.

• In the beginning you are not in the DHT network, you need someone to introduce you, any one in DHT can. Generally we can send Find_node requests to router.bittorrent.com:6881, dht.transmissionbt.com:6881 and so on, and then our DHT can start working.

KRPC implementation You can refer to here.

Summarize

DHT as a whole is a hash table, first of all we are inside a node, we send to others krpc Find_node or get_peers message, is to perform a get (key) operation on this hash table. Sending a announce_peer message to someone is doing a set (key, Val) operation on the hash table.

At last

HTTPS://GITHUB.COM/SHIYANHUI/DHT complete code here, like this article, to a star on GitHub:)

Http://bthub.io is a BT seed search engine written based on the sniffer above.

Any questions can be asked here: Https://github.com/shiyanhui/dht/issues

OK, here today, about how to download, we will say next.

You can follow my public number and get the next push in time.