Bitcask Storage Model

Bitcask Storage Model

Bitcask is a log-type, hash-table-based Key-value storage model, with Riak and beansdb new versions of K-V systems Bitcask as storage models.

journaled Data Storage

What is a journaled type? is append only, all writes are appended without modifying the old data, just like our various server logs. In the Bitcask model, the data files are written to the file with a log type, and the file has a certain size limit, when the file size is increased to the corresponding limit, a new file will be generated, the old file will be read-only and not write. At any point in time, only one file is writable, in the Bitcask model it is called the active data file, and the other files that have reached the limit size are called older data file, such as:

The data structure in the file is very simple and is a single piece of data write operation, each piece of data is structured as follows:

The above data items are the size of the Key,value,key, the size of value, the timestamp (which should be), and the CRC checksum for the previous items. (The data delete operation does not delete the old entry, but instead sets value to a special value for marking)
Data files are successive strips of data in the above format, such as:

The above is a log-type data file, but the file continues to persist, it will certainly be infinite expansion, in order to solve the problem, and other log storage System Bitcask also has a regular merge operation.
The merge operation, which periodically scans all older data file for a time and generates a new data file (not including the active data file because it is still being written), The merge is actually the deletion of multiple operations on the same key with the principle of keeping only the newest one. After each merge, the newly generated data file will no longer have redundant data.

In fact, there are two types of files in Bitcask:

• XXX.W: That is, the above-mentioned data file, XXX is a value, write full 1G after the new file, the value of the increment, the program to the maximum value of a file Append data;
• Write.pos: Records the current value of the largest XXX.W file (Fileno), as well as the file's write offset (offset), convenient for writing data at the time of fast positioning;

After defining the data file and writing the file for write displacement, we implement the data writing in the following ways:

• Check the current XXX.W file write offset + whether the length of data currently being written is greater than max_file_size (that is, the maximum SIZE of a data file, which we define as 1G), or if greater than create a. w file with an ordinal increment of 1;
• Call the open, write function to append the data to the end of the current. w file;
• Update the value of Fileno, Offset in Write.pos;

index data based on hash table

The write operation has a Write.pos file indicating the Fileno and Offset to be written, but for a read operation, how quickly can you find the file and file displacement of a key corresponding to the data in the vast data? This depends on the hash table data index in memory, such as

This structure of the hash table includes three information for locating the data value, which is the file ID number (file_id), the position of value in the file (Value_pos), the size of the value (VALUE_SZ), so we read the File_ The ID corresponds to the file's Value_pos beginning with the VALUE_SZ byte, and we get the value we need. The entire process is as follows:

Because of the existence of a hash table, our write operation needs to update a piece of content, that is, the corresponding relationship of the hash table. A write operation requires a sequential disk write and a memory operation.

In addition, because the index hash table is stored in memory, each time the process restarts need to rebuild the hash table, which requires the entire scan all the data files, if the data file is large, this will be a very time-consuming process. So the Bitcask model contains a section called hint file, which is designed to improve the speed of rebuilding the hash table .

The new data file is generated when the merge operation is performed in the old data file, and the Bitcask model actually encourages the generation of a hint file, which is the structure of each item in the hint file that is very similar to the data structure in The difference is that he does not store the specific value, but rather the location of the value, so that when rebuilding the hash table, you do not need to scan all of the data file files, but simply need to read and rebuild the rows of the hint file in a row. Greatly improves the speed of restarting the database with data files.

Bitcask Storage Model