Linux C Development-command parsing of memcached source Analysis (2)

Objective

From our previous chapter "Linux C Development-memcached source analysis based on the Libevent network model " We have a basic understanding of the Memcached network model. In this chapter, we need to interpret Memcached's command parsing in detail.

We review the previous chapter and found that memcached is divided into the main thread and n worker threads. The main thread is primarily used to listen for socket connections for ACCPET clients, while worker threads are primarily used to take over specific client connections.

Between the main thread and the worker thread mainly through the libevent-based pipe read and write events to listen to, when there is a connection to practice, the main thread will connect to a certain worker thread to take over, the late client and the service side of the read and write work will be in this worker thread.

The worker thread is also based on the Libevent event, which triggers the event's callback function when a read or write event comes in.

So how does memcached parse the command data messages uploaded by the client? We will explain in detail below.

Memcached's command parsing source code analysis
Starting from Drive_machine

We saw the read-write event callback function for the client connection in the previous section: Event_handler, the final call in this method is Drive_machine.

void Event_handler (const int FD, const short which, void *arg) {conn *c;//assembly conn struct C = (conn *) Arg;assert (c! = NULL); c-> ; which = which;/* sanity */if (fd! = C->SFD) {if (Settings.verbose > 0) fprintf (stderr, "catastrophic:event FD doesn ' t match Conn fd!\n '); Conn_close (c); return;} The Drive_machine is finally forwarded to the method Drive_machine (c);/* Wait for next event */return;}

Conn Data structure

Each connection will have its own conn data structure. This structure primarily stores the basic information for each connection.

Some of the more important parameters used in this chapter are:

char * rbuf: used to store commands in client data messages.

int rsize:the size of the rbuf.

char * Rcurr: The character pointer of the unresolved command.

int rbytes: The length of the command to parse.

typedef struct CONN Conn;struct conn {int sfd;    Sasl_conn_t *sasl_conn;    BOOL authenticated;    Enum Conn_states State;    Enum Bin_substates substate;    rel_time_t Last_cmd_time;    struct event event;    Short ev_flags;   short which;   /** which events were just triggered */char *rbuf;  /** buffer to read commands into */char *rcurr;   /** But if we parsed some already, the is where we stopped */int rsize;  /** Total Allocated size of RBUF */int rbytes;    /** how much data, starting from Rcur, does we have unparsed */char *wbuf;    Char *wcurr;    int wsize;    int wbytes;    /** which state to go in after finishing current write */enum conn_states write_and_go; void *write_and_free;  /** Free the memory after finishing writing */char *ritem;    /** when we read in an item's value, it goes here */int rlbytes; /* Data for the nread state *//** * Item is used to hold an item structure created AFTER reading the command * line of SET/ADD/REPLACE commands, but before we finished reading the actual * data.     The data is read in Item_data (ITEM) to avoid extra copying.     */void *item;    /* FOR Commands Set/add/replace */* data for the Swallow state */int sbytes;    /* How many bytes to swallow */* data for the mwrite state */struct IOVEC *iov;   int iovsize;   /* Number of elements allocated in iov[] */int iovused;    /* Number of elements used in iov[] */struct MSGHDR *msglist;   int msgsize;   /* Number of elements allocated in msglist[] */int msgused;   /* Number of elements used in msglist[] */int msgcurr;  /* element in msglist[] being transmitted now */int msgbytes;   /* Number of bytes in current MSG */item **ilist;    /* List of items to write out */int isize;    Item **icurr;    int ileft;    Char **suffixlist;    int suffixsize;    Char **suffixcurr; int suffixleft;   Enum protocol Protocol; /* Which protocol this connection speaks */enum network_transport transport; /* What transport are used by this connection */* data for UDP clients */int request_id; /* Incoming UDP request ID, if this is a UDP "connection" */struct SOCKADDR_IN6 request_addr;    /* Udp:who sent the most recent request */socklen_t request_addr_size; unsigned char *hdrbuf;   /* UDP packet headers */int hdrsize;   /* Number of headers ' worth of space is allocated */bool noreply; /* True If the reply should not being sent.        */* Current STATS command */struct {char *buffer;        size_t size;    size_t offset;    } stats;     /* Binary Protocol Stuff *//* This is where the binary header goes */Protocol_binary_request_header binary_header; uint64_t CAs; /* The CAS to return */short cmd;    /* Current command being processed */int opaque;    int Keylen;     Conn *next; /* Used for generating a list of ConnStructures */Libevent_thread *thread; /* Pointer to the thread object serving this connection */};

Drive_machine:

Drive_machine in this method, all the logic that needs to be handled is judged by c->state .

Conn_listening: Listening status

Conn_waiting: Wait Status

Conn_read: Read status

Conn_parse_cmd: command-line parsing

static void Drive_machine (conn *c) {bool stop = False;int sfd;socklen_t addrlen;struct sockaddr_storage addr;int nreqs = s Ettings.reqs_per_event;int Res;const char *str; #ifdef have_accept4static int use_accept4 = 1; #elsestatic int USE_ACCEPT4 = 0; #endifassert (c! = NULL); while (!stop) {switch (c->state) {case conn_listening://.... More Code}

Let's continue to look at the code for the Conn_read, conn_wait, and Conn_parse_cmd states.

1. When the client has a data message, it triggers the Conn_read case.

2. Conn_read will go to read the socket data, if not read the data, then will call conn_waiting this case, will continue to wait for the client data message escalation.

3. If an unexpected error occurs in the Conn_read, the Conn_close case is called and the client connection is closed.

4. If conn_read reads the data, or if it has data in its own buf, it will parse the command and invoke the Conn_parse_cmd case.

This way is to continue waiting for the client's data message to arrive case Conn_waiting:if (!update_event (c, Ev_read | Ev_persist) {if (Settings.verbose > 0) fprintf (stderr, "couldn ' t update event\n"); Conn_set_state (c, conn_closing); break;} During the wait process, the connection state is set to read state, and stop is set to true to exit the while (stop) loop conn_set_state (c, conn_read); stop = true;break;//event that reads the data, When the client has a data message upload, it will trigger the Libevent Read event case Conn_read://try_read_network mainly read the TCP data//return Try_read_result enum type structure, through this enumeration type, To determine if the data has been read, read failure, and so on res = IS_UDP (c->transport)? TRY_READ_UDP (c): Try_read_network (c); switch (res) {//did not read the data, then continue to set the event to wait. while (stop) will continue to loop, go to call conn_waiting this casecase read_no_data_received:conn_set_state (c, conn_waiting); break;// If there is data to read, this time need to call Conn_parse_cmd logic//conn_parse_cmd: Mainly used to parse the Read command case read_data_received:conn_set_state (c, Conn_ PARSE_CMD); break;//read failed state, direct call conn_closing Close Client connection case Read_error:conn_set_state (c, conn_closing); break;case Read_memory_error:/* Failed to allocate + MEMORY *//* state already set by Try_read_network */break;} break;//, this is the client that resolves memcached.-side commands, such as parsing: the set username Zhulicase Conn_parse_cmd://try_read_command method is critical to read the command//If this method returns to 0, the parse command fails (because of the reason for the TCP sticky packet unpacking, may not complete the command, need to continue to wait for data arrival) if (Try_read_command (c) = = 0) {/* wee need more data! *///this side of the comment seems to be wrong, it should be we need more data!conn_set_st Ate (c, conn_waiting);} Break

Try_read_network

This approach is primarily to read TCP network data. The data that is read is put into the c->rbuf buf.

If BUF has no space to store more data, it triggers a memory block reallocation. Redistribution, memcached limit of 4 times, is estimated to be concerned about the client's attack caused by the storage of command-line data packets BUF constant Ralloc.

This method reads the command data passed by the client via TCP, static enum Try_read_result try_read_network (conn *c) {//This method will eventually return the enumerated type of Try_read_result Default setting Read_no_data_received: No data is received for enum Try_read_result gotdata = read_no_data_received;int Res;int num_allocs = 0; ASSERT (c! = NULL);//c->rcurr store unresolved command content pointer c->rbytes How many unresolved data//c->rbuf are used to read the buf of the command, to store pointers to the command string c->rsize Rbu F's Sizeif (c->rcurr! = c->rbuf) {if (c->rbytes! = 0)/* Otherwise there ' s nothing to copy */memmove (C->rbuf, C- >rcurr, c->rbytes); c->rcurr = C->rbuf;} Loop reads data from FD while (1) {//If the BUF is full, you need to reassign a larger piece of memory//when the unresolved data size is greater than or equal to the size of the BUF block, you will need to reassign if (C->rbytes >= c->rsize {//assign up to 4 times if (Num_allocs = = 4) {return gotdata;} ++num_allocs;//new block of memory, memory size of rsize twice times char *new_rbuf = realloc (C->rbuf, c->rsize * 2); if (!new_rbuf) {Stats_ LOCK (); stats.malloc_fails++; Stats_unlock (); if (Settings.verbose > 0) {fprintf (stderr, "couldn ' t realloc input buffer\n");} c->rbytes = 0; /* Ignore what we read */out_of_memory (c, "Server_eRror out of memory reading request "); c->write_and_go = Conn_closing;return read_memory_error;} C->rcurr and C->rbuf point to the new buf block C->rcurr = C->rbuf = New_rbuf;c->rsize *= 2; Rsize is multiplied by 2}//avail to calculate the amount of space remaining in the buf block int avail = c->rsize-c->rbytes;//This way we can see the socket read method//c-> The Id//c->rbuf + c->rbytes of the SFD socket means that the newly read data is stored from the spare memory address in the BUF block//avail the maximum read data per receive, res = read (C->SFD, C- >rbuf + c->rbytes, avail);//If the received result res is greater than 0, then the data is read//set to the Read_data_received enumeration type in the socket, indicating that the data is read if (res > 0) {Pthread_mutex_lock (&c->thread->stats.mutex);//thread lock C->thread->stats.bytes_read + = Res;pthread_ Mutex_unlock (&c->thread->stats.mutex); gotdata = read_data_received;c->rbytes + = res; Amount of data not processed + the command currently read to Sizeif (res = = avail) {continue;} else {break;}} Determine the two cases of read failure if (res = = 0) {return read_error;} if (res = =-1) {if (errno = = Eagain | | errno = = ewouldblock) {break;} return read_error;}} return gotdata;}

Try_read_command

This method is mainly used to read the commands in the rbuf.

Example command: Set username zhuli\r\n get username \ n

The command in the data message is separated by the newline character. Because data packets have sticky and unpacking characteristics, they can only be parsed until the command line is complete. All only matches the \ n symbol to match a complete command.

If we already have a command line in C->rbuf that we can handle, we can call this function to handle the command resolution static int Try_read_command (conn *c) {assert (c! = NULL); assert (c-> Rcurr <= (c->rbuf + c->rsize)); Assert assert here (c->rbytes > 0); if (C->protocol = = Negotiating_prot | | c->transport = = udp_transport) {if (Unsig Ned char) c->rbuf[0] = = (unsigned char) protocol_binary_req) {c->protocol = Binary_prot;} else {c->protocol = ASC Ii_prot;} if (Settings.verbose > 1) {fprintf (stderr, "%d:client using the%s protocol\n", C->sfd,prot_text (C->protocol)); }}//There are two modes, whether binary mode or ASCII mode if (C->protocol = = Binary_prot) {//More code} else {//This side mainly handles non-binary mode command parsing char *el, *cont;//if C- >rbytes==0 indicates that there are no command messages that can be processed in the BUF container, the return 0//0 is to allow the program to continue waiting to receive the new client message if (C->rbytes = = 0) return 0;//find command with \n,memcache command through \ n to split//When the client's data message comes up, memcached to determine whether the received command packet is complete//such as command: Set username 10234344 \ n get username \n//by looking for a newline character in the received data This command can be split into two commands, set and get command//el return \ n character pointer address el = MEMCHR (C->rcurr, ' \ n ', c->rbytes);//If no \ n is found, the command is incomplete, then 0 is returned. Continue waiting to receive new customersEnd data message if (!el) {//c->rbytes is the length of the packet received//This is very interesting, if the data packet received more than 1K, then memcached go back to determine whether the request is too large, is there a problem? The link to this client will then be closed if (C->rbytes > 1024x768) {/* * * We didn ' t have a ' \ n ' in the first K. This _has_ to is a * large multiget, if not we should just nuke the connection. */char *ptr = c->rcurr;while (*ptr = = ") {/* ignore leading whitespaces */++ptr;} if (Ptr-c->rcurr > 100| | (STRNCMP (PTR, "get", 4) && strncmp (PTR, "gets", 5))) {Conn_set_state (c, conn_closing); return 1;}} return 0;} If \ n is found, the complete command in C-&GT;RCURR is cont = el + 1; The next command starts the pointer node//This side to determine whether it is \ r \ n, if it is \ r \ n, then El moves forward an if ((El-c->rcurr) > 1 && * (el-1) = = ' \ r ') {el--;} The last character of the command is then separated by the (string ending symbol) *el = ' + '; ASSERT (cont <= (C->rcurr + c->rbytes)); c->last_cmd_time = Current_ Time Last command time//processing command, C->rcurr is command Process_command (c, C->rcurr); c->rbytes-= (Cont-c->rcurr); Why isn't this place written like this? c->rbytes = C->rcurr-contc->rcurr = cont; Point C->rcurr to the next command pointer node assert (C->rcurr <=(C->rbuf + c->rsize));} return 1;}

Process_command

This method is mainly used to handle specific commands. After the command is decomposed, it is distributed to different specific operations.

In the command handler///previous method, we found the character \ n in rbuf and then replaced it with \0static void Process_command (conn *c, char *command) {//tokens structure, this way will c-> Rcurr (command) commands are split//and the command is separated into multiple elements by a space symbol//For example: Set username Zhuli, then split into 3 elements, set and username and Zhuli//max_ respectively The Tokens maximum value is 8, which describes the memcached command line, which can be split up into 8 elements token_t tokens[max_tokens];size_t ntokens;int Comm;assert (c! = NULL); Memcached_process_command_start (C-&GT;SFD, C->rcurr, c->rbytes); if (Settings.verbose > 1) fprintf (stderr, " <%d%s\n ", C-&GT;SFD, command);/* * For commands Set/add/replace, we build a item and read the data * directly into it , then continue in Nread_complete (). */c->msgcurr = 0;c->msgused = 0;c->iovused = 0;if (ADD_MSGHDR (c)! = 0) {out_of_memory (c, "Server_error out of Mem Ory preparing response "); return;} Tokenize_command is very important, the main thing is to split the command//and put the split command elements into the tokens array//Parameters: Command for commands ntokens = tokenize_command (Command, tokens, max_tokens);//tokens[command_token] command_token=0//The first parameter of the broken command is the action method if (Ntokens >= 3&& ( strcmp (Tokens[command_tokEn].value, "get") = = 0) | | (strcmp (Tokens[command_token].value, "bget") = = 0))) {//Handle get command Process_get_command (c, tokens, Ntokens, false);} else if ((Ntokens = = 6 | | ntokens = = 7) && ((strcmp (Tokens[command_token].value, "add") = = 0 && (comm =nrea D_add)) | | (strcmp (Tokens[command_token].value, "set") = = 0&& (comm = nread_set)) | | (strcmp (Tokens[command_token].value, "replace") = = 0&& (comm = nread_replace)) | | (strcmp (Tokens[command_token].value, "prepend") = = 0&& (comm = nread_prepend)) | | (strcmp (Tokens[command_token].value, "append") = = 0&& (comm = nread_append))) {//Process update command Process_update_command (c, tokens, Ntokens, comm, FALSE);//More code ....}

Tokenize_command:

This method is primarily used to break down commands. In particular, a command statement is decomposed into multiple elements.

Example: Set username zhuli\n

is decomposed into three elements: set and username and Zhuli three elements.

Split command method Static size_t Tokenize_command (char *command, token_t *tokens,const size_t max_tokens) {char *s, *e;size_t Ntoken s = 0; Command parameter cursor size_t len = strlen (command); command length unsigned int i = 0;assert (Command! = NULL && tokens! = null && max_tokens > 1); s = e = Command;fo R (i = 0; i < len; i++) {//The pointer keeps going, if a space is encountered, it will stop, split the command element, and put it into the tokens array if (*e = = ") {if (s! = e) {Tokens[ntokens].val UE = S;tokens[ntokens].length = e-s;ntokens++;//This side of the space to replace the \0//memcached side of the code is very good, this side of the command to cut, and did not copy the memory block, Instead, the original memory block is cut *e = ' + ';//up to 8 elements if (Ntokens = = max_tokens-1) {e++;s = e;/* So we don ' t add a extra token */break;}} s = e + 1;} e++;} if (s! = e) {tokens[ntokens].value = S;tokens[ntokens].length = e-s;ntokens++;} /* * If We scanned the whole string, the terminal value pointer is null, * otherwise it is the first unprocessed character . */tokens[ntokens].value = *e = = ' + '? Null:e;tokens[ntokens].length = 0;ntokens++;//return value is the number of arguments, for example, 3 elements are decomposed, then return 3return Ntokens;}

Process_get_command

Example of a GET command:

command to process get request static inline void Process_get_command (conn *c, token_t *tokens, size_t ntokens,bool return_cas) {// Handle get command char *key;size_t nkey;int i = 0;item *it;//&tokens[0] is the method of operation//&tokens[1] for key//token_t Stored value and lengthtoken_t *key_token = &tokens[key_token];char *suffix;assert (c! = NULL);d o {//If key is not 0while long (key_ Token->length! = 0) {key = Key_token->value;nkey = key_token->length;//Determines if the length of the key exceeds the maximum length, memcache The maximum length of the key is 250//this place needs to be very careful, we in the ordinary use, still need to notice the byte length of the key of if (Nkey > Key_max_length) {//out_string to the external output data out_string (c, " Client_error bad command line format "), while (i--> 0) {item_remove (* (c->ilist + i));} return;} This is the way to get data from Memcached's memory storage. It = Item_get (key, Nkey), if (settings.detail_enabled) {//Status record, key record Count method Stats_prefix_ Record_get (Key, Nkey, NULL! = it);} If you get the data if (it) {//c->ilist holds the buf//for writing data externally if the IList is too small, reallocate a piece of memory if (I >= c->isize) {Item **new_list = ReAlloc ( C->ilist,sizeof (item *) * C->isize * 2); if (new_list) {c->isize *= 2;c->ilist = new_list;} else {stats_lock (); stats.malloc_fails++; Stats_unlock (); Item_remove (it); break;}} /* * Construct the response. Each hits adds three elements to the * outgoing data list: * "VALUE" * key * "" + flags + "+ data length +" \ r \ n "+ data (with \ r \ n) *///initialization returns the data structure if (RETURN_CAS) {//More code ...} /* * If the command string hasn ' t been fully processed, get the next set * of tokens. *///if the command line commands are not all processed, continue to the next command//command line, you can get multiple elements if (key_token->value! = NULL) {ntokens = Tokenize_command (key_ Token->value, tokens, max_tokens); key_token = Tokens;}} while (key_token->value! = NULL); C->icurr = C->ilist;c->ileft = I;if (return_cas) {C->suffixcurr = C->s Uffixlist;c->suffixleft = i;} if (Settings.verbose > 1) fprintf (stderr, ">%d end\n", C->SFD);/* If the loop was terminated because of Out-of-mem Ory, it's not reliable to add end\r\n to the buffer, because it might isn't END in \ r \ n. So we send server_error instead. */if (Key_token->value! = NULL | | Add_iov (c, "end\r\n", 5)! = 0| | (IS_UDP (C->transport) && build_udp_headers (c)! = 0)) {Out_of_memory (c, "Server_error out of memory writing get Response");} else {conn_set_state (c, conn_mwrite); c->msgcurr = 0;}}

Linux C Development-command parsing of memcached source Analysis (2)