Redis Source Code Analysis (12)---redis-check-dump Local database detection

This document I in today's analysis study, has had a kind of indefinitely feeling, the code quantity 700+ the code, finally opens to the system is a process () method. Here is said that the database detection, is for key detection, will be used, the following mentioned structure:

/* Data type to hold opcode with optional key name an success status *//* used for key detection, subsequent detection operations are used entry struct */typedef struct {//key's name    char* key;    The type    int type;    Whether it is a success state    char success;} entry;

A lot of the subsequent APIs are related to this struct, the final detection of this code is actually a file called Dump.rdb, after the detection is also added to the cyclic redundancy check CRC64. The following highlights the API:

int Checktype (unsigned char t)/* Whenever a new obj type is added, it is necessary to detect whether the type is reasonable */int readbytes (void *target, long num)/* reads the num from the current file offset position  byte position */int processheader (void)/* Reads the header of the snapshot file, detects if the header name or version number is correct */int loadtype (entry *e)/* For entry assign obj type */int peektype () /* Pop-up version number */int processtime (int type)/* Remove bytes used to represent time */uint32_t loadlength (int *isencoded)/* Sub-type read length */char *loadintege Robject (int enctype)/* Gets the numeric value as a character by encoding the current integer, returns */char* Loadlzfstringobject ()/* for the extracted string */char* loadstringobject ()/ * Gets the current file information string Object */int Processstringobject (char** Store)/* To assign the string object to the passed in Parameter */double* loaddoublevalue ()/* file read double type value * /int Processdoublevalue (double** store)/* Give parameters to the double type */int loadpair (entry *e)/* Read key value pair */entry loadentry ()/* Get ENT RY Key structure */void printcentered (int indent, int width, char* body)/* Output interface symmetric information */void printvalid (uint64_t ops, uint64_t by TES)/* Output valid information */void printskipped (uint64_t bytes, uint64_t offset)/* Output skipped skip bytes bytes Information */void printerrorstack (entry * e)/* Output error stack information */void procesThe S (void)/* Process method is the primary method of performing the detection */ 

Method inside a lot of loadxxx () method, these load method is indeed more useful, in this detection file, the writer also very humanized constructs the error structure, is used to simulate the wrong information stack output.

/* Hold a stack of errors *//* error message struct */typedef struct {//Specific error message string    char error[16][1024];    Internal offset    size_t offset[16];    Error message rank    size_t level;} errors_t;static errors_t errors;

Different level levels correspond to different error messages. In the API there is a more critical approach, loadentry, to obtain key-related structures;

/* Get entry key struct */entry loadentry () {entry E = {NULL,-1, 0};    uint32_t length, offset[4];    /* Reset Error container */errors.level = 0;    Offset[0] = Curr_offset;    Here the value of type if (!loadtype (&e)) {return e;    } offset[1] = Curr_offset; if (E.type = = Redis_selectdb) {if (length = Loadlength (NULL)) = = Redis_rdb_lenerr) {shift_error (OFFSE            T[1], "Error reading database Number");        return e;            } if (length > N) {shift_error (offset[1], "Database number out of range (%d)", length);        return e; }} else if (E.type = = redis_eof) {if (Positions[level].offset < positions[level].size) {Shift_e        Rror (Offset[0], "unexpected EOF");        } else {e.success = 1;    } return E;            } else {/* Optionally consume expire */if (E.type = = Redis_expiretime | | E.type = = Redis_expiretime_ms) {if (!processtime (e.type)) return e;        if (!loadtype (&e)) return e;        } offset[1] = Curr_offset; Call Loadpair to Entry assignment key if (!loadpair (&e)) {Shift_error (offset[1], "ERROR for type%s", Types[e.typ            E]);        return e; }}/* All entries is followed by a valid type: * e.g. a new entry, Selectdb, EXPIRE, EOF */offset[2] = CUR    R_offset;        if (peektype () = =-1) {Shift_error (offset[2], "followed by invalid type");        Shift_error (Offset[0], "ERROR for type%s", Types[e.type]);    e.success = 0;    } else {e.success = 1; } return E;

One of the key assignments inside of Key,value in the Loadpair () method:

/* Read key value pair */int Loadpair (entry *e) {uint32_t offset = curr_offset;    uint32_t i;    /* Read Key First *///Read key value from file char *key;    if (Processstringobject (&key)) {E->key = key;        } else {Shift_error (offset, "ERROR reading entry key");    return 0;    } uint32_t length = 0;        if (E->type = = Redis_list | |        E->type = = Redis_set | |        E->type = = Redis_zset | | E->type = = Redis_hash) {if (length = Loadlength (NULL)) = = Redis_rdb_lenerr) {shift_error (offset, "            Error reading%s Length ", Types[e->type]);        return 0; }}//reads the value values followed by the key value switch (e->type) {case Redis_string:case redis_hash_zipmap:case REDIS_LIST_ZIPL Ist:case redis_set_intset:case redis_zset_ziplist:case redis_hash_ziplist://Because a struct like ziplist,zipmap is actually a node.          Connected to a super string, so is directly read if (!processstringobject (NULL)) {Shift_error (offset, "ERROR reading entry value");  return 0;    } break; Case Redis_list:case Redis_set://And the above 2 types are the traditional structure, to divide the node to read for (i = 0; i < length; i++) {OFFSE            t = Curr_offset; if (!processstringobject (NULL)) {Shift_error (offset, "ERROR reading element at index%d (length:%d)", I,                length);            return 0;    }} break;            Case redis_zset:for (i = 0; i < length; i++) {offset = Curr_offset;  if (!processstringobject (NULL)) {Shift_error (offset, "ERROR reading element key at index%d (length:%d)",                I, length);            return 0;            } offset = Curr_offset; if (!processdoublevalue (NULL)) {Shift_error (offset, "ERROR reading element value at index%d (length:%d)"                , I, length);            return 0;    }} break;            Case redis_hash:for (i = 0; i < length; i++) {offset = Curr_offset; if (!proCessstringobject (NULL)) {Shift_error (offset, "ERROR reading element key at index%d (length:%d)", I, Leng                TH);            return 0;            } offset = Curr_offset; if (!processstringobject (NULL)) {Shift_error (offset, "ERROR reading element value at index%d (length:%d)                ", I, length);            return 0;    }} break;        Default:shift_error (offset, "Type not Implemented");    return 0;    }/* Because we ' re done, we assume success *////As long as executed, we are determined to be successful e->success = 1; return 1;}

If the e-success=1 that the key detection will pass. Why do you say that, let's see the main detection method process () method:

/* The process method is the primary method for performing the detection */void process (void) {uint64_t num_errors = 0, Num_valid_ops = 0, num_valid_bytes = 0;    Entry entry;    Read file header get snapshot file version number int dump_version = Processheader (); /* Exclude The final checksum for RDB >= 5. 'll is checked at the end. */if (dump_version >= 5) {if (Positions[0].size < 8) {printf ("RDB version >= 5 but no roo            M for checksum.\n ");        Exit (1);    } positions[0].size-= 8;    } level = 1;        while (Positions[0].offset < positions[0].size) {positions[1] = positions[0];        Entry = Loadentry ();            if (!entry.success) {//If entry is not a success status Printvalid (Num_valid_ops, num_valid_bytes);            Printerrorstack (&entry);            num_errors++;            Num_valid_ops = 0;            num_valid_bytes = 0;            /* Search for Next valid entry */uint64_t offset = positions[0].offset + 1;                        int i = 0; Then look for the back 3 valid ENtries while (!entry.success && offset < positions[0].size) {Positions[1].offset = O                Ffset;                    /* Find 3 consecutive valid entries *//Find 3 valid entries for (i = 0; i < 3; i++) {                    Entry = Loadentry ();                if (!entry.success) break; }/* Check if we found 3 consecutive valid entries */if (I < 3) {off                set++;  }}/* Print how many bytes we have skipped to find a new valid opcode * * if (offset <            Positions[0].size) {printskipped (offset-positions[0].offset, offset);        } Positions[0].offset = offset;            } else {num_valid_ops++;            Num_valid_bytes + = Positions[1].offset-positions[0].offset;        /* Advance position */positions[0] = positions[1];    } free (Entry.key);   } /* Because there is another potential error, * print How many valid OPS we have processed */printvalid (num_valid_    OPS, num_valid_bytes); /* Expect an EOF */if (entry.type! = redis_eof) {/* Last byte should is EOF, add error */Errors.level        = 0;        Shift_error (Positions[0].offset, "expected EOF, got%s", Types[entry.type]);        /* This is a EOF error so reset type */Entry.type =-1;        Printerrorstack (&entry);    num_errors++; }/* Verify checksum *//version number >=5, verify checksum if (dump_version >= 5) {uint64_t CRC = CRC64 (0,position        S[0].data,positions[0].size);        uint64_t Crc2;        unsigned char *p = (unsigned char*) positions[0].data+positions[0].size;               Crc2 = ((uint64_t) p[0] << 0) |               ((uint64_t) p[1] << 8) |               ((uint64_t) p[2] << 16) |               ((uint64_t) p[3] << 24) |               ((uint64_t) p[4] << 32) | ((uint64_t) p[5] << 40) |               ((uint64_t) p[6] << 48) |        ((uint64_t) p[7] << 56);        if (CRC = Crc2) {shift_error (Positions[0].offset, "RDB CRC64 does not match.");        } else {printf ("CRC64 checksum is ok\n");        }}/* Print summary on Errors */if (num_errors) {printf ("\ n");    printf ("Total unprocessable opcodes:%llu\n", (unsigned long long) num_errors); }}

If you want to know the detailed principles of testing, it may be helpful to know the structure of Dump.rdb's file content beforehand.

Redis Source Code Analysis (12)---redis-check-dump Local database detection