LEVELDB Source Analysis--Log

We know that in a database system in order to ensure the reliability of data, we will record the operation of the system log. The function of the log is to restore the data when the system is down, so the log system is extremely important for a storage system that requires reliability. Next we analyze Leveldb log, first in the LEVELDB source directory has doc/log_format.txt, this file detailed description of the LEVELDB log format:

Record: =    checksum:uint32    //  crc32c of type and data[]; Little-endian    Length: UInt16       //  Little-endian    type:uint8          // one of Full,first, middle, last     Data:uint8[length]

such as (Image citation):

Leveldb when writing logs, the log file is divided into a number of 32K of file blocks, each read and write the log in each of these 32K units. The approximate composition of the log file for leveldb after such processing can be considered as follows:

So let's take a brief look at log writes:

When you want to write an article first to determine if the current block is sufficient to store the log

S1. If it is sufficiently straightforward to install the format to write;

S2. If not enough then calculate how much content can be stored outside the head, assemble the content into First's log Typpe write, and then take a new block to determine if it is sufficient to store the remaining log data

While (data is not finished)

S21. If sufficient, the assembly is written as last form;

S22. If it's still not enough, the assembly is written as middle

So it's easy to understand the Full,first,middle and the last:

Full: A complete log is written to block

First: A log, but the current block cannot be fully written, some data is written to the next block, the current block of data is only the beginning of the log (first) section

Middle: The log content is followed by the continuation of the last log in the previous block, and this block is not fully finished, and the data in the next block continues to have the log

Last: The final part of the previous block's unfinished log;

It is also important to note that according to the previous description we can imagine a block after writing a portion of the data will be left some space, the space may be greater than 7byte, equal to 7byte, less than 7byte; Why should we use 7byte as the demarcation line? The log header (Crc|length|type) length is 7, if more than 7 can save at least one fisrt part of the log record, and equal to 7 just save a header, less than 7 even the header is not saved. Log is based on this reason, less than 7 o'clock to add "\", 7 to save an empty header. Let's take a look at the code logic

Status Writer::addrecord (const slice& Slice) {

BOOL true;

// Loop writes to the log file until you finish writing

 Do {    Const intleftover = kblocksize-block_offset_; if(Leftover <kheadersize) {//  less than 7 byte (header size) padding 0x0        if(Leftover >0) {Dest_->append (Slice ("\x00\x00\x00\x00\x00\x00", leftover)); } block_offset_=0; }    Constsize_t avail = kblocksize-block_offset_-kheadersize; Constsize_t Fragment_length = (left < avail)?Left:avail;    RecordType type; Const BOOLEnd = (left = =fragment_length); {//  Determine if this paragraph can be completed     if(Begin &&end) { //  start end in this block type=Kfulltype; } Else if(BEGIN) { //  start at this block, end not in type=Kfirsttype; } Else if(end) { //  end at, start not in type=Klasttype; } Else{ //  start end is not at this block type=Kmiddletype; } //  encode and then write to the file s=Emitphysicalrecord (Type, PTR, fragment_length); PTR+=fragment_length; Left-=fragment_length; Begin=false; }  while(S.ok () && left >0); returns;}

The code of the

reads the log in log_reader.cc, the code logic is responsible for much more than write, the main is to read when will need to add a lot of error processing related content, the specific code is not listed, understand the format of the log file is easy to read. Of course, at the same time, the logic of the exception handling is also the material we learn, understand how the master of the various error handling.