LevelDB compaction operation

"LevelDB compaction Operation "

For Leveldb, the write record operation is simple, and deleting a record just writing a delete tag is done, but the read record is complex and needs to be found in the memory and in each level file in order of freshness, at a high cost. In order to speed up the reading speed,Leveldb took a compaction way to collate the existing records, in this way, to remove some no longer valid KV data, reduce the size of the data, reduce the number of files and so on .

The compaction mechanism and process of leveldb is basically consistent with bigtable, with three types of Compaction:minor, major and full in bigtable. The so-called minor compaction, is the memtable in the data exported to the Sstable file, major compaction is to merge different levels of sstable files, and the full Compaction is to merge all the sstable.

Leveldb contains two of them, minor and Major.

First look at the process of minor compaction. The purpose of Minor compaction is to save the contents to a disk file when the memtable size in memory is a certain value, and figure 8.1 is its mechanism.

　　

As can be seen from 8.1, when the number of memtable to a certain extent will be converted to immutable memtable, at this time cannot write to the record, only the KV content can be read from. Previously introduced, immutable Memtable is actually a multi-level queue skiplist, where the records are sorted by key. So this minor compaction implementation is also very simple, that is, according to immutable Memtable recorded from small to large traversal, and then write to a level 0 of the new sstable file, after writing the index data to establish the file, This completes a minor compaction. Also can be seen, for the deleted records , in the minor compaction process does not really delete this record , the reason is very simple, here only know to delete the key record, but where the KV data? That requires complex lookups, So in the minor compaction do not delete, but the key as a record to write to the file, as for the real delete operation, in the later higher level of compaction will do.

When the number of sstable files in a level exceeds a certain set value, Leveldb selects a file (level>0) from the sstable of the levels and merges it with level+1 sstable files of a higher hierarchy, which is major Compaction.

We know that at levels greater than 0, the keys within each sstable file are stored in small to large order, and the key range between the different files (the minimum key and the maximum key within the file) does not overlap. Level 0 sstable files are special, although each file is also based on key from small to large, but because level 0 files are generated directly through minor compaction, so any two level 0 under two sstable files may overlap on the key range. So when doing major compaction, for levels greater than level 0, select one of the files on the line, but for 0, after specifying a file, it is likely that there are other Sstable file's key range and this file overlap, In this case, to find all overlapping files and level 1 files to merge, that is, level 0 in the file selection, there may be more than one file to participate in major compaction.

Leveldb after selecting a level for compaction, but also to choose which file to do compaction,leveldb here is a small trick, that is, take turns, for example, this time is file a compaction, then the next time is the key The range is compaction to file B next to file A, so that each file will have the opportunity to merge with the top level files in turn.

If you choose the level L file A and levels l+1 layer of files to merge, then the problem comes again, you should choose what files to be l+1 to merge? LEVELDB Select all the files in the l+1 layer and file a that overlap on the key range to merge with file a.

That is, the file A of level L is selected, and then all the files that need to be merged are found in level l+1 b,c,d ... etc... The remaining question is how does the major merger work? That is, given a series of files, each file is a key in order, how to merge these files, so that the newly generated file is still a key in order, while throwing away what is no longer valuable kv data.

Figure 8.2 illustrates this process.

　　

The process of Major compaction is as follows: for multiple files in a multi-merge sort, find the smallest key record in turn, that is, to reorder all the records in multiple files. Then take certain criteria to determine whether the key still needs to be saved, if the judgment does not save value, then directly throw away, if you feel you need to continue to save, then write it to the level l+1 layer in a newly generated sstable file. In this way, the KV data one by one processing, formed a series of new l+1 layer data files, the previous L-layer file and l+1 layer to participate in the compaction file data at this time has no meaning , so all deleted. This completes the merge process of the L-layer and l+1-layer file records.

So in the process of major compaction, what is the criterion for judging whether a KV record is discarded? One of the criteria is that for a key, if the key is present in a lower-than-l layer, the KV can be thrown away during the major compaction process. As we have previously analyzed, if there is a record of the same key in a file with a level lower than L, then there is a more fresh value for key, then the past value is meaningless, so it can be deleted.

Reference: http://www.cnblogs.com/haippy/archive/2011/12/04/2276064.html

　　

LevelDB compaction operation