LEVELDB: A fast and lightweight Key-value repository (translated)

Original: http://leveldb.googlecode.com/svn/trunk/doc/index.html

Translator: phylips@bmy 2011-8-16

Translation: http://duanple.blog.163.com/blog/static/70971767201171705113636/

The LEVELDB library provides a permanent key value store. The key and value are arbitrary byte sequences. In this key value storage system, key is sorted according to the user-declared comparison function.

Open a Database

A LEVELDB database has a file system directory name associated with it. All the contents of the database are stored in the directory. The following example shows how to open a database, or how to create one when necessary:

#include

#include "Leveldb/db.h"

LEVELDB::D b* DB;

Leveldb::options Options;

Options.create_if_missing = true;

Leveldb::status Status = leveldb::D b::open (Options, "/tmp/testdb", &db);

ASSERT (Status.ok ());

...

If you want the above code to cause an error if the database already exists, you need to add the following line before the open call:

Options.error_if_exists = true;

State (status)

You may have noticed the Leveldb::status data type above. This type of value is returned by the vast majority of function calls that can be faulted in leveldb. You can also print out some error messages in case of an error

Leveldb::status s = ...;

if (!s.ok ()) Cerr << s.tostring () << Endl;

Close Database

After you have finished processing a database, you can delete the database object directly.

... open the DB as described above ...

... do something with db ...

Delete db;

Read operations and write operations

The database provides the put, Delete, and get methods to modify/query the database. For example, the following code saves the value stored under Key1 under Key2.

std::string value;

Leveldb::status s = db->get (Leveldb::readoptions (), Key1, &value);

if (S.ok ()) s = Db->put (Leveldb::writeoptions (), key2, value);

if (S.ok ()) s = Db->delete (Leveldb::writeoptions (), key1);

Atomic Update

It is to be noted that. In the above operation, if the process dies before the delete key1 after the put Key2, the same value value may be stored under multiple key values. You can avoid such a problem by performing a set of update operations using the Writebatch class atomicity:

#include "leveldb/write_batch.h"

...

std::string value;

Leveldb::status s = db->get (Leveldb::readoptions (), Key1, &value);

if (S.ok ()) {

Leveldb::writebatch Batch;

Batch. Delete (Key1);

Batch. Put (key2, value);

s = Db->write (Leveldb::writeoptions (), &batch);

}

Writebatch

Will hold a series of changes to the database, which will be executed in the order in which they were added. In addition to providing the assurance of this atomicity, Writebatch can also accelerate their execution by placing multiple updates in the same batch and in the presence of a large number of update operations.

Synchronous write operations

By default, writes to LEVELDB are asynchronous: They are returned after the write is passed from the process to the operating system. The transmission from the operating system memory space to the underlying persistent storage device is asynchronous. You can turn on the sync flag to allow a write operation to wait until the data is actually written to a permanent store before returning. (In a POSIX system, this is done by calling Fsync (...) or fdatasync (...) or Msync (..., ms_sync) before the write operation returns)

Leveldb::writeoptions write_options;

Write_options.sync = true;

Db->put (write_options, ...);

Asynchronous write operations are typically 1000 times times faster than synchronization. The disadvantage is that when the machine crash off, it may lose some of the last updates. It is important to note that if only the crash of the write process (for example, not reboot) will not cause any loss, because even if sync=false, an update operation must upload the data from the application space to the system space before it is complete.

Asynchronous write operations are generally safe to use. For example, when loading a large amount of data into a database, you can process the missing updates by reloading them after crash. You can also use a blending mode, such as every n write operation, to do a synchronization, so that when the crash occurs, you only need to start from the last sync to restart it (only need to let the synchronization of the write operation to update a restart from where to start the tag).

Writebatch can also provide an improvement for asynchronous write operations. Multiple update operations can be placed to the same writebatch and then use a synchronous write operation (such as Write_options.sync=true). This extra synchronization overhead can be allocated to multiple write operations.

Concurrent

Only one process can open the database at the same time. To prevent misuse, the LEVELDB implementation will request a lock from the operating system. Within a process, the same
leveldb::D B
objects can be safely shared by multiple concurrent threads. For example, multiple threads can write data in the same database, obtain iterators, perform get calls without requiring additional synchronization (the LEVELDB implementation automatically completes the required synchronization). However, other objects, such as iterators and Writebatch, may require additional synchronization. If two threads share the same object, they must use their own lock mechanism to protect access to such objects.

Iterations

The following example shows how to print out all key value pairs in a database.

leveldb::iterator* it = Db->newiterator (Leveldb::readoptions ());

For (It->seektofirst (); It->valid (); It->next ()) {

cout << It->key (). ToString () << ":" << it->value (). ToString () << Endl;

}

Check for any errors found during the scan

Delete it;

The following example shows how to handle only the data within a given key value boundary [Start,limit]:

For (It->seek (start);

It->valid () && It->key (). ToString () < limit;

It->next ()) {

...

}

can also be processed in reverse order (may be slower than sequential processing)

For (It->seektolast (); It->valid (); It->prev ()) {

...

}

Snapshots

Snapshots provides a read-only view of the consistency of the entire Key-value storage state. You can specify that you want to read from a particular version of a database by setting Readoptions::snapshot to a non-null value. If its value is empty, the current state is read by default. Snapshots are typically created by the Db::getsnapshot () method:

Leveldb::readoptions options;

Options.snapshot = Db->getsnapshot ();

... apply some updates to db ...

leveldb::iterator* iter = db->newiterator (options);

... read using ITER to view the "state" when the "snapshot" was created ...

Delete iter;

Db->releasesnapshot (Options.snapshot);

leveldb::snapshot* Snapshot;

Leveldb::writeoptions write_options;

Write_options.post_write_snapshot = &snapshot;

Leveldb::status Status = Db->write (write_options, ...);

... perform other mutations to DB ...

Leveldb::readoptions read_options;

Read_options.snapshot = snapshot;

leveldb::iterator* iter = Db->newiterator (read_options);

... read as of the "state just" Write call returned ...

Delete iter;

Db->releasesnapshot (snapshot);

The return values of the It->key () and It->value () calls above are leveldb::slice types. Slice is a simple structure that contains a pointer to an array of external bytes and a length. Returning slice is less expensive than returning a std::string type because we do not have to copy those large key and value values. In addition, the Leveldb method cannot return a null-terminated C-style string because its key and value are allowed to contain '.

C + + string and null-terminated C-style strings can be simply converted to a slice type:

Leveldb::slice S1 = "Hello";

std::string str ("World");

Leveldb::slice s2 = str;

A slice type can also be simply converted to a C + + String type:

std::string str = s1. ToString ();

assert (str = = std::string ("Hello"));

Using slices requires extra care because it relies on the caller to ensure the validity of the external byte array that slice points to. For example, the following code is problematic:

Leveldb::slice Slice;

if (...) {

std::string str = ...;

slice = str;

}

Use (slice);

Because the scope of the IF statement has ended, STR will be destructor, so that the slice point to the space does not exist.

Comparison device

The preceding example uses the default sort function for key, sorted in dictionary order. You can provide a custom comparer when you open the database. For example, assuming that the key of the database is composed of two digits, we first sort by the first number, if the equality compares the second one. First you need to define a subclass of Leveldb::comparator that satisfies the following rules:

Class Twopartcomparator:public Leveldb::comparator {

Public

Three-way comparison function:

If a < b:negative result

If a > b:positive result

Else:zero result

int Compare (const leveldb::slice& A, const leveldb::slice& b) Const {

int A1, A2, B1, B2;

Parsekey (A, &a1, &A2);

Parsekey (b, &B1, &B2);

if (A1 < B1) return-1;

if (A1 > B1) return +1;

if (A2 < B2) return-1;

if (A2 > B2) return +1;

return 0;

}

Ignore the following methods for now:

Const char* Name () const {return "Twopartcomparator";}

void Findshortestseparator (std::string*, const leveldb::slice&) const {}

void Findshortsuccessor (std::string*) const {}

};

Now use a custom comparer to create a database:

Twopartcomparator CMP;

LEVELDB::D b* DB;