A preliminary understanding of leveldb-in depth leveldb

Article Reference http://blog.chinaunix.net/uid-26575352-id-3245476.html

1. Leveldb Introduction

Leveldb is a standalone version of the K-V storage system implemented by Google's two-bit engineers with the following features

1. Both key and value are arbitrary byte arrays that support memory and persist storage

2. Data are sorted by key

3. Users can override sorting functions

4. Contains basic data operation interface, Put (Key,value), Get (key), Delete (key)

5. Multiple operations can be treated as an atomic operation

6. Users can generate snapshot, so that read operations are not affected by the write operation, the reading process to see the final data consistency

7. Support the operation of the iterator to the data

8. Data use snappy automatic compression

9. External operations (such as file system operations, etc.) are used by a virtual interface that allows users to customize the operating system accordingly

2. LEVELDB Limitations

1. LEVELDB non-relational database, SQL query not supported or index not supported

2. Support only single process (multi-threaded) access db at the same time

3. Client-server model not supported, users need to package themselves

3. LEVELDB Basic Framework

LEVELDB is essentially a set of storage systems and some operating interfaces that are available on this storage system. To facilitate understanding of the entire system and its processing flow, we can look at Levledb from two different angles: static angle and dynamic angle. From a static point of view, it can be assumed that the entire system is running (constantly inserting delete read data), at this time we give leveldb photography, from the photos can see how the system's data in memory and disk is how the distribution, in what state, etc. from a dynamic point of view, mainly to understand how the system is writing a record, Read a record, delete a record, but also include in addition to these interface operations, such as internal operations such as compaction, the system after the crash, how to restore the system and so on

Leveldb as a storage system, the basic framework is as follows in the entire system operation

, the storage medium of leveldb is divided into two kinds: memory and disk. There are memtable and immutable memtable in memory, and there are log files, manifest files, current files, and sstable files of the level in the disk;

1). When the user inserts a Key-value data into the DB, the log file is written and the current record is written to memtable when the log is successfully written. Why write a log file when writing data, mainly because the newly inserted data will be saved in memory, in order to prevent the system crashes caused by the new insertion data loss, so write the log file to ensure that after the landing, and then write memory. This way, even if the system crashes, it can be recovered from log.

2). The data in memtable is readable and writable when the amount of data memtable reaches a data volume. The current memtable becomes immutable memtable, and the read-only cannot be modified. Regenerate the new memtable and log files, and the new data is written to the new log and memtable.

3). Immutable memtable data will be dump to the disk sstable file, the sstable file on the disk is hierarchical, the first layer level0 to the nth layer Leveln ..., each level has a lot of sstable files, Each file is sorted by key. Note that level0 is not the same as other level, the key of the sstable file in Level0 can be duplicated, and the key of other level sstable file is guaranteed not to be duplicated.

4). Because there are many sstable files in each level, each sstable file has a key range. Therefore, a file is required to save the key range of Sstabel in the current level. Manifest files are primarily used to store information about the sstable in each level

5). As the system continues to run, the sstable files in each level may become more and more, and this time the DB will automatically merge the sstable files at the same level or different level. This time manifest will change, so we need a current file to record the latest manifest file.

Not to be continued ...

A preliminary understanding of leveldb-in depth leveldb