Key/value Storage System of log structure bitcask

Minor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast;
MSO-Hansi-theme-Font: minor-Fareast; color: #333333 "> bitcaskmso-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font:
Minor-Fareast; color: #333333 "> from Riak, It is a log-structured storage system. It is used to store the underlying key/value of the Riak distributed database.

Minor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast;
MSO-Hansi-theme-Font: minor-Fareast; MSO-bidi-font-family:; MSO-font-kerning:
0pt ">

10.5pt; font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:
; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
MSO-bidi-font-family:; MSO-font-kerning: 0pt "> bitcaskFont-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
Basic Features of MSO-bidi-font-family:; MSO-font-kerning: 0pt ">:

MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font:
Minor-Fareast; MSO-bidi-font-family:; MSO-bidi-theme-Font: minor-Fareast;
MSO-font-kerning: 0pt "> 1.
MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font:
Minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast "> key/value; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast "> stores logs in sequence and can only append (append-only) Key/value writes. Each write operation is sequential. When the value corresponding to a key changes, the new key/value is appended to the end of the file. Minor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast;
MSO-Hansi-theme-Font: minor-Fareast; MSO-bidi-font-family:; MSO-font-kerning:
0pt ">

MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font:
Minor-Fareast; MSO-bidi-font-family:; MSO-bidi-theme-Font: minor-Fareast;
MSO-font-kerning: 0pt "> 2.
Minor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast;
MSO-Hansi-theme-Font: minor-Fareast "> the hash table stores the corresponding relationship (INDEX) between the key and its physical storage location, and the read operation time complexity is 0 (1 ); you can make the index persistent to reduce the time for re-indexing during restart. Minor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast;
MSO-Hansi-theme-Font: minor-Fareast; MSO-bidi-font-family:; MSO-font-kerning:
0pt ">

MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font:
Minor-Fareast; MSO-bidi-font-family:; MSO-bidi-theme-Font: minor-Fareast;
MSO-font-kerning: 0pt "> 3.
Minor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast;
MSO-Hansi-theme-Font: minor-Fareast "> after the system has been running for a period of time, you can perform the merge operation to merge the same keys to free up space. 10.5pt; font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:
; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
MSO-bidi-font-family:; MSO-font-kerning: 0pt ">

MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font:
Minor-Fareast; MSO-bidi-font-family:; MSO-bidi-theme-Font: minor-Fareast;
MSO-font-kerning: 0pt "> 4.
Font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
Color: #333333 "> the write operation only requires one disk seek. The read operation first searches for the index, obtains the location and size of the value corresponding to the key, and then reads the value data, two disk seek operations are required. Therefore, the read/write efficiency is quite high. Minor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast;
MSO-Hansi-theme-Font: minor-Fareast; MSO-bidi-font-family:; MSO-font-kerning:
0pt ">

Font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
Color: #333333 ">

MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font:
Minor-Fareast; color: #333333 "> bitcaskMinor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast;
MSO-Hansi-theme-Font: minor-Fareast; color: #333333 "> Data Storage

MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font:
Minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast; color: #333333 "> In bitcask, each key/value pair is stored in the structure shown in, here, CRC is the verification code of the Data body, tstamp is the timestamp, ksz is the length of the key, valuesz is the length of the value, followed by the key and value.

Font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
Color: #333333 ">

Font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
Color: #333333 ">

Font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
Color: #333333 "> key/valuefont-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
Color: #333333 "> store data in the preceding format, minor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast;
MSO-Hansi-theme-Font: minor-Fareast "> when a key/value is added, bitcask directly appends it to the end of the data file, even if the corresponding key already exists in the data file, this may cause bitcask to store multiple versions corresponding to a key. To save storage space, bitcask will perform regular merge operations, merge multiple operations on the same key. After each merge operation, no redundant data exists in the data file.

Font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
Color: #333333 ">

MSO; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast "> index data based on Hash Tables

MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font:
Minor-Fareast "> log-type data files make write operations very fast (one of the advantages of log-type is that the disk is regarded as a tape, and the sequential read/write efficiency is very high ), however, if key value search is performed on such log data, the efficiency will be very low. bitcask uses the hash table in the memory to create an index of the key and its storage location, this enables efficient read operations. Each entry in the hash table is as follows, which stores the size and offset of the value corresponding to the key.

MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font:
Minor-Fareast ">

Font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
Color: #333333 ">

MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font:
Minor-Fareast; color: #333333 "> hint FileMinor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast;
MSO-Hansi-theme-Font: minor-Fareast; color: #333333 "> accelerate index Reconstruction

Font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
Color: #333333 "> bitcask; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font:
Minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast; color: #333333 "> indexes are stored in the memory.; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast "> if we do not do any additional work, when the system starts to recreate the hash table, you need to scan the entire data file. If the data file is large, it will be a very time-consuming process. The bitcask model contains a part called hint file, which aims to improve the speed of rebuilding hash tables.

MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font:
Minor-Fareast "> each key/value pair corresponds to an entry in the hint file. This entry mainly stores the correspondence between the key and the value size and position.
The file will be persisted to the hard disk. Every time the system starts, you only need to read the hint file in sequence to quickly construct the hash index table.

MSO-Fareast-font-family:; MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font:
Minor-Fareast; color: #333333 ">

MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:; MSO-Fareast-theme-Font:
Minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast; color: #333333 "> references:

Font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
Color: #333333 "> http://downloads.basho.com/papers/bitcask-intro.pdf

Font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast;
Color: #333333 "> http://lgone.com/html/y2010/861.html

Font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast "> http://www.unclejoey.com /? Tag = bitcask

Font-family:; MSO-ascii-theme-Font: minor-Fareast; MSO-Fareast-font-family:;
MSO-Fareast-theme-Font: minor-Fareast; MSO-Hansi-theme-Font: minor-Fareast ">