Memcached learning notes-storage command source code analysis part I: memcached learning notes

Memcached learning notes-storage command source code analysis part I: memcached learning notes

For Original article reprinted, please mark it. Thank you.

The previous article analyzed the memcached connection model and learned how memcached efficiently processes client connections. This article analyzed the process_update_command function in the memcached source code and explored the set command of the memcached client, how memcached parses client text commands, analyzes memcached memory management, and how the LRU algorithm works.

Parse client text commands

The client sends the set operation to the memcached server. The memcached server reads the Client Command and the connection status of the client is changed from conn_read> conn_parse_cmd. At this time, the memcached server starts parsing the command. Memcached server calls the try_read_command function parsing command. memcached receives commands in two formats: Binary and text ).

1 static int try_read_command (conn * c) {2 3 //.......... 4 5 if (c-> protocol = binary_prot) {6 7 // binary format 8 //.... 9 10} else {11 char * el, * cont; 12 13 // if no command is received from the client, return to the conn_waiting status, wait for more client data 14 if (c-> rbytes = 0) 15 return 0; 16 17 el = memchr (c-> rcurr, '\ n ', c-> rbytes); 18 if (! El) {19 if (c-> rbytes> 1024) {20/* 21 * We didn't have a' \ n' in the first k. this _ has _ to be a22 * large multiget, if not we shoshould just nuke the connection.23 */24 char * ptr = c-> rcurr; 25 while (* ptr = '') {/* ignore leading whitespaces */26 + ptr; 27} 28 29 if (ptr-c-> rcurr> 100 | 30 (strncmp (ptr, "get", 4) & strncmp (ptr, "gets ", 5) {31 32 conn_set_state (c, conn_closing); 33 return 1; 34} 35} 36 37 return 0; 38} 39 40 // The client message ends with '\ r \ n'. 41 cont = el + 1; 42 if (el-c-> rcurr)> 1 & * (el-1) = '\ R') {43 el --; 44} 45 * el =' \ 0 '; 46 47 assert (cont <= (c-> rcurr + c-> rbytes); 48 49 // 50 process_command (c, c-> rcurr) where the command is actually parsed ); 51 52 c-> rbytes-= (cont-c-> rcurr); 53 c-> rcurr = cont; 54 55 assert (c-> rcurr <= (c-> rbuf + c-> rsize); 56} 57 58 return 1; 59}

Before analyzing the process_command function, let's look at the memcached command format:

1 <command name> <key> <flags> <exptime> <bytes> [noreply] \ r \ n 2 3 cas <key> <flags> <exptime> <bytes> <cas unique> [noreply] \ r \ n 4 5 // For example, set command: 6 set key 0 60 2 7 12 8 STORED 9 10 // space corresponds to space 11 set => <command name> 12 key => <key> 13 0 => <flags> 14 60 => <exptime> 15 2 => <bytes>

Memcached calls the tokenize_command function in process_command to process the command according to the preceding command format, and stores the fields in the corresponding location of token_t * tokens.

1 // parameter 1: string of the command 2 // parameter 2: After parsing the command, store the struct array of each field of the command 3 // parameter 3: maximum number of command fields 4/* 5 * tokens [0] => <command name> information 6 * tokens [1] => <key> information 7 * tokens [2] => <flags> information 8 */9 static size_t tokenize_command (char * command, token_t * tokens, const size_t max_tokens) {10 char * s, * e; 11 size_t ntokens = 0; 12 size_t len = strlen (command); 13 unsigned int I = 0; 14 15 assert (command! = NULL & tokens! = NULL & max_tokens> 1); 16 17 s = e = command; 18 for (I = 0; I <len; I ++) {19 if (* e = '') {20 if (s! = E) {21 tokens [ntokens]. value = s; // value stores the string values of each field, for example, 'set' 22 tokens [ntokens]. length = e-s; // length indicates the length of each field. For example, the length of 'set' is 323 ntokens ++; 24 * e = '\ 0 '; 25 if (ntokens = max_tokens-1) {26 e ++; 27 s = e;/* so we don't add an extra token */28 break; 29} 30} 31 s = e + 1; 32} 33 e ++; 34} 35 36 if (s! = E) {37 tokens [ntokens]. value = s; 38 tokens [ntokens]. length = e-s; 39 ntokens ++; 40} 41 42/* 43 * If we scanned the whole string, the terminal value pointer is null, 44 * otherwise it is the first unprocessed character.45 */46 tokens [ntokens]. value = * e = '\ 0 '? NULL: e; 47 tokens [ntokens]. length = 0; 48 ntokens ++; 49 50 return ntokens; 51}

After parsing the text command, return to the process_command function. We can see the familiar command. Yes, next, in the multi-branch judgment of an if-else, memcached uses tokens [COMMAND_TOKEN]. value decides to call the function to process the corresponding command:

1 static void process_command (conn * c, char * command) {2 3 //.... 4 5 ntokens = tokenize_command (command, tokens, MAX_TOKENS); 6 if (ntokens> = 3 & 7 (strcmp (tokens [COMMAND_TOKEN]. value, "get") = 0) | 8 (strcmp (tokens [COMMAND_TOKEN]. value, "bget") = 0) {9 10 // here is the branch 11 process_get_command (c, tokens, ntokens, false) that executes the get command ); 12 13} else if (ntokens = 6 | ntokens = 7) & 14 (strcmp (tokens [COMMAND_TOKEN]. value, "add") = 0 & (comm = NREAD_ADD) | 15 (strcmp (tokens [COMMAND_TOKEN]. value, "set") = 0 & (comm = NREAD_SET) | 16 (strcmp (tokens [COMMAND_TOKEN]. value, "replace") = 0 & (comm = NREAD_REPLACE) | 17 (strcmp (tokens [COMMAND_TOKEN]. value, "prepend") = 0 & (comm = NREAD_PREPEND) | 18 (strcmp (tokens [COMMAND_TOKEN]. value, "append") = 0 & (comm = NREAD_APPEND )))) {19 20 // here is the branch 21 process_update_command (c, tokens, ntokens, comm, false) that executes the set, add, replace, and other commands ); 22 23} else if (ntokens = 7 | ntokens = 8) & (strcmp (tokens [COMMAND_TOKEN]. value, "cas") = 0 & (comm = NREAD_CAS) {24 25 // The process_update_command function is also executed here, which also performs write operations on the corresponding key, unlike the previous branch, the last parameter is true, which means that the CAS protocol is used for the write process, 26 // CAS aims to ensure consistency during concurrent writing. 27 process_update_command (c, tokens, ntokens, comm, true ); 28 29} else if .............

Memcached storage command Analysis

Memcached splits the memory into chunks of various sizes and divides chunks of the same size into groups. Each chunk set is called slab. The memory allocation of Memcached is in the unit of Page. The default Page value is 1 MB. You can specify it by using the-I parameter at startup. Slab consists of multiple pages, which are cut into multiple chunks according to the specified size.

Each pair of [key, value] data is encapsulated into the item struct. Each type of slab uses an item linked list to maintain all its items. For example, the data size of an item plus the header information of the item (for easy description, the sum of the two items, collectively referred to as [key, value], is 90KB, the chunk block size of slab [I] Is 136KB, the chunk block size of slab [I-1] Is 88KB, the item will be assigned a chunk block of slab [I] (and saved to an item linked list maintained by slab [I]), so as to minimize memory fragments.