Write a search engine (0x04)---B + tree with Golang

This is a creation in Article, where the information may have evolved or changed.

This article is longer and more boring, please keep your patience to read.

In the first two chapters, we introduce the two storage structures of inverted index and inverted index dictionary, namely jumping table and hash table, this article introduces another data structure, he is also used extensively in the field of information retrieval, I github On the implementation of the search engine dictionary is also used in this data structure, it is a B + tree.

First, we look at what is a tree, the tree is a very basic data structure in the program design, remember the University of data structures, lists, stacks, queues, and then is the tree, although at that time presumably everyone was pre-sequence traversal, the sequence traversal, post-order traversal toss over, but the tree is really a very useful data structure.

As we said in the first column of table 2, the main problem is to quickly find the corresponding word, and then find the corresponding word of the inverted table, in addition to jumping tables and hash table, B + tree can satisfy the condition, the B + tree is a variant of the second tree, we do not look at the second tree, interested people can go directly to Google, We are mainly talking about the B + Tree, is a 3-storey B + tree, I painted it may be found out a little bit in and out, but it does not matter, the key B + tree This data structure of the idea of everyone understand the line.

Suppose we have a set of numbers 34,40,67,5,37,12,45,24, so that's what it looks like to save them as a B + tree.

We obviously see a few features

• The size of each node is 2

• The last element of the last node of the non-leaf layer is NULL

• The lowest-level leaf nodes are arranged sequentially, this example is from small to large

• Each element of the inner node above points to the largest of the next node in the same number

I try to put a B + tree simple point, online information, search books, look very complicated, first we look at how to build this tree, I try to use the figure, less text or understand a bit, ahead of a large number of map warning.

First of all, our array is 34,12,5,67,37,40,45,24

The first step is to initialize the B + tree.

At this time, nothing, but occupied two nodes, identified as none , indicating that the element is meaningless, marked NULL for Infinity

The second step, insert 34 This element, then the diagram becomes such a child

We see that the insertion process follows the pointer all the way to the leaf node, discovers that the leaf node is empty, then inserts the element into the leaf node's head, then returns to the previous level node, moves null back, and then puts the first element at the maximum of his child node , remember this sentence: To the maximum value of his child node

Step three, then insert the second element 12

This step is a little more complicated

• Starting from the root node, we find that one element of the 12 small root node is "Here is the 1th element" and down the pointer.

• To reach the leaf node, found that 12 is less than the leaf node of an element, the description can be placed in the leaf node, and the leaf node has an empty position, then directly to the 12 in the order of size inserted into this node

Fourth step, then insert 5

This step is a little more complicated, resulting in a split

• Traverse from the root node, 5 is less than 34, follow the pointer down and reach the leaf node

• Reached the leaf node, found that 5 is less than the leaf node of an element, the description can be placed in the leaf node, but the node is full, then, split a new node, put 5 into the old node, the squeezed elements moved to the new node

• To return to the previous node, since the largest element of the first leaf node has become 12, the element of that node is changed from 34 to the largest element of the leaf node that points to 12

• Because the new generation becomes a node, the null element points to the newly generated node

Fifth Step, then we insert 67

This step is relatively simple

• Traverse from the root node, 67 is less than NULL, go down the pointer and reach the leaf node

• Reached the leaf node, found that 67 is greater than each element of the node, and the leaf node has a vacancy, directly inserted can

Sixth step, we insert 37, after this I will not write, interested can draw their own

This step is complicated, this step is not only split, but split two times, and the number of layers increased by one layer

• Traverse from the root node, 37 is less than NULL, go down the pointer and reach the leaf node

• Reached the leaf node, 37 is less than 67 of the leaf node, indicating that it can be inserted into this node, but the node is full, we follow the fourth step of the operation, split node.

• After splitting, produced a [34,37], a [67, no] two nodes, go up, found that the previous layer of the node inserted 37 is also full, continue to follow the fourth step split.

• After splitting, found that the upper layer has no nodes, then create a new root node when the upper node, according to the split step to the root node assignment.

According to these six steps, the first 5 elements are inserted into the B + tree, the following steps you can walk a walk, B + Tree basic idea is such a son, perhaps I did not follow the textbook approach, but this does not affect everyone's understanding, I believe after reading although you have no standard algorithm steps in mind, But there should be a rough outline, just need to think carefully about the steps.

In general, the insertion step of the B + tree is no more than the following steps

• Start each time from the root node

• Compare size, find the element less than the current value, follow the pointer down, continue to compare size, always reach the leaf node, then this leaf node is the node you want to operate.

• In the leaf node only a few operations, one is the leaf node has empty position, then directly inserted into the leaf node is full, then split a node out.

• No matter in the leaf node to do that kind of operation, and finally to follow the pointer back, if there is no division, then the upper layer will not split, may update the value of the upper node element, if split, then with two split nodes up and down, the update value is updated value, the split split.

• If one layer splits to the top, then add a root node.

The find operation is almost the same as the update operation, which is the first two steps of the update operation, not to mention.

The general update is the first to find, find the leaf node, and then update, and then follow the pointer up to continue to split, the next step up the general situation is the first thought of the two-way pointer, but the two-way pointer split when a bit of trouble, need to put two pointers are new nodes, I realized when the use of a stack, Find the leaf node when the past node pressed the stack, to reach the leaf node, the completion of the insertion operation, the top of the loop when the stack bounced out on the line, less a pointer.

Go back to the first column of the table 2, if it is that table, with this B + tree plus inverted chain, the final data structure will grow like this (the size of the string I casually write, the Chinese order of the elder brother's brain not out, you think they are small to large order it)

Well, at this point, an inverted index is set up, composed of two parts, I realized the time is so realized, a structure with B + Tree storage dictionary, the other is a sequential file, B + Tree leaf node has a file offset to the inverted file, of course, you can also use the previous hash table or jumping tables, There are even other types of trees, such as trie trees, or you can have other new, efficient data structures.

Let's talk about the B + tree, why choose it?

I used to use a hash table before, and most of the search engines use a hash table, why use a tree?

• First of all, in order to save space, if a hash table, if there is a field is the primary key, and is irregular (such as Cookieid), then if the huge amount of documents, the hash table bucket will be very large, will be very memory, and I debug the machine is 8G memory of the Mac.

• Second, look at the hash table, the time complexity of the query is O (1), it looks really good, if it is a full-text search engine, because key is a string, and the basic is the Chinese string, the whole Chinese vocabulary is hundreds of thousands of, really good, but if the field is not necessarily Chinese word of things, There are some other things, such as various IDs, because it is a generic search, so does not give specific fields to define a special hash function, so it may be a large collision, the complexity is not O (1), if it is a specific scene of the search, to circumvent this problem, according to their own business needs, You can even use the perfect hash function, and I realized it mostly to be more generic, so I used a B + tree.

• We look at the B + tree itself, if we can store 100 elements per node, then a 4-tier B + Tree, you can store 100 million data, whether it is a primary key field or other fields are enough, and a 4-tier B + tree retrieval, need to traverse 4 nodes, each node with two points to find, is log100 (2 for the bottom), about 7 times, 4 stories, the worst need to query 28 times, if it is 3 layer, the worst 21 times, although and the hash is so much slower than that, but 1 cycles about 4 CPU clock cycle bar, for the current server computer, even 21 cycles + Condition judgment is also microsecond level, feel not too out of the difference, not to mention impossible every time is so point back, have to check 21 times it.

• Again, my index generation is generated by the segment, followed by the integration of multiple segments of the index, if it is a B + tree, the dictionary is sequential, you see the above diagram, the leaf node is a pointer linked together, so the merging segment can be used when a multi-merge will be merged, if the hash, because it is not sequential , merging needs to be hashed again, more troublesome.

• Also, B + tree This data structure is very suitable for disk retrieval, only need to set the size of each node to a disk page size (generally 4K, as to why the size of the page, and the structure of mechanical hard disk and the pre-reading mechanism, interested can check their own, but now are SSDs, The impact of this is not very large), the pointer to the page number of the disk, then do not load into the memory, directly on the disk can be retrieved, especially for the large amount of data dictionary (such as the primary key), index database index (such as MySQL Innedb) is basically a B + tree implementation, If you are interested, you can open a single article to say this.

• Finally, B + trees are stored sequentially, so they can be searched (although I don't use them), and the hash table can only be searched in strict order.

Finally say I realize this B + tree, first, in order to occupy less memory, I am using the form of the disk to achieve, and use a mmap way to speed up read and write, no two-way pointer, and the stack to record the path of the query, speed is OK, construct a tree of 100,000 random string about 3 seconds, A random query of 100,000 times takes approximately 150 milliseconds, 1.5 microseconds at a time.

Of course, I realized the time is relatively hasty, is according to the algorithm hard coding fast, so I this B + tree also has a very large optimization space, first of all, my key is now OK, not more than 64 bytes, and a maximum of 100 elements per node, at that time, in order to fast, determine the key and the number of elements better programming , if it becomes dynamic more space-saving, second, there is no special consideration of the continuous key case, the insertion of continuous key will cause a waste of space half, also, the speed problem to mmap to solve, if the memory is enough, actually start when the pre-read non-leaf node to memory, the query will be faster, But at present basically satisfies the demand, everybody if is interested in the B + tree realization, may look at the bolt this project, this is a B + tree realizes the Kvdb, but also has the transaction the OH.

OK, this one is written here, the weekend to go out to play, next week to continue ...

Finally, you are welcome to scan the following public number subscriptions, which will be sent here first:)