Leveldb Simple Introduction and principle--essence: Like Nedb, insert data file growing (snapshot), and then delete old data to do update

Transferred from: http://www.cnblogs.com/haippy/archive/2011/12/04/2276064.html

Have time to look at the whole article!

Speaking of LEVELDB may not be clear to you, but if you are an IT engineer and do not know the following two major God-level engineers, then your leader will be unable to hold: Jeff Dean and Sanjay Ghemawat. These two are Google's heavyweight engineers and few Google Fellow.

Jeff Dean's man:http://. Google. COM/people/Jeff/index. HTML, Google's large-scale distributed platform BigTable and MapReduce are primarily designed and implemented by.

Sanjay Ghemawat People:http://. Google. COM/people/Sanjay/index. HTML, Google massively distributed platform GFS, BigTable and MapReduce primarily design and implement engineers.

Leveldb is an open source project initiated by these two great-god-level engineers, in short,leveldb is a C + + library capable of handling 1 billion-scale Key-value data persistence storage . As described above, these two bits are designed and implemented by BigTable, and if you understand bigtable, you should know that there are two core parts in this far-reaching distributed storage System: Master Server and tablet server. Where master Server does some storage of management data and distributed scheduling, the actual distributed data storage and read and write operations are done by the tablet server, and LEVELDB can be understood as a simplified version of the tablet server.

Leveldb has some of the following features:

First,leveldb is a persistent storage kv system, unlike Redis's memory-type KV system, LEVELDB does not eat as much memory as Redis, but instead stores most of the data on disk.

Second, when storing data, levledb is stored in order according to the key value of the record, that is, the adjacent key value is stored sequentially in the stored file, and the application can customize the key size comparison function, and Levledb stores the records sequentially according to the user-defined comparison function.

Again, like most KV systems, the Leveldb interface is simple, with basic operations such as writing records, Reading Records, and deleting records. Atomic bulk operations for multiple operations are also supported.

In addition, theLEVELDB supports the data snapshot (snapshot) feature so that read operations are not affected by write operations , and consistent data can always be seen during a read operation.

In addition to this, LEVELDB supports operations such asdata compression , which can be directly helpful in reducing storage space and increasing the efficiency of the IO.

Leveldb performance is very prominent, the official website reported that its random write performance of 400,000 records per second, and random read performance of 60,000 records per second . In general, Leveldb writes are much faster than read operations, while sequential read and write operations are much faster than random read and write operations. As to why this is the case, after watching our follow-up leveldb, it is estimated that you will understand its intrinsic reasons.

LEVELDB is essentially a set of storage systems and some operating interfaces that are available on this storage system. To facilitate understanding of the entire system and its processing flow, we can look at Levledb from two different angles: static angle and dynamic angle. From a static point of view, it can be assumed that the entire system is running (constantly inserting delete read data), at this time we give leveldb photography, from the photos can see how the system's data in memory and disk is how the distribution, in what state, etc. from a dynamic point of view, mainly to understand how the system is writing a record, Read a record, delete a record, but also include in addition to these interface operations, such as internal operations such as compaction, the system after the crash how to restore the system and so on.

The overall architecture described in this section is mainly from the static point of view, then the following sections will detail the static structure involved in the file or memory data structure, LEVELDB in the second half of the day to introduce the dynamic perspective of the leveldb, that is how the whole system is working.

Leveldb as a storage system, the data record of the storage media including memory and disk files, if as mentioned above, when the leveldb run for a while, we give LEVELDB perspective, then you will see the following scenes:

Figure 1.1:LEVELDB Structure

As can be seen, the composition of the LEVELDB static structure consists of six parts, in-memory memtable and immutable memtable, the current file on disk, the log file, the manifest file, and the Sstable file. the storage flow is as follows:

• When inserting a key-value data, LEVELDB first inserts the data into the log file (append), and then writes the memtable in a successful way, guaranteeing both efficient writing and stability of the data.
• When the data inserted by Memtable has reached a boundary, it is converted to immutable memtable, which is supported by the new memtable. At the same time, the LEVELDB in the background will be immutable memtable through the scheduler Dump to the sstable file on disk.
• The data inside the sstable is key-ordered. By immutable memtable constantly dump out of the sstable file more and more, will be the compact operation, the formation of a new level of sstable files.

Reference: http://www.cnblogs.com/haippy/archive/2011/12/04/2276064.html

Leveldb Simple Introduction and principle--essence: Like Nedb, insert data file growing (snapshot), and then delete old data to do update