Socket connection and dispatch of distributed cache system Memcached State machine

The previous section has analyzed that the main thread listens for socket registration events and the callback function in the worker thread that connects the socket registration event is event_handler, and the core part of Event_handler is a finite state machine: Drive_machine. Therefore, the specific business processing of this state machine will be analyzed in depth.

The memcached encapsulates each socket as a conn structure that contains the file descriptor of the socket, SFD, the Register events event, the connection state structure conn_states, and many other information fields, including the state structure: conn_ The various states of the socket are included in the States. The state machine Drive_machine is the state structure to determine the current state of the socket, so that the business logic processing.

The structure of the connection state is as follows:

The structure of the possible state of the socket
Enum Conn_states {
Conn_listening,//monitor status/**< the socket which listens for connections */
Conn_new_cmd,//prepare for the next connection/**< Prepare connection for next command */
Conn_waiting,//waiting to read a packet/**< waiting for a readable socket */
Conn_read,//Read network data/**< reading in a command line */
Conn_parse_cmd,//parse buffer data/**< try to parse a command from the input buffer */
Conn_write,//Simple reply data/**< writing out a easy response */
Conn_nread,//Read fixed byte of network data/**< reading in a fixed number of bytes */
Conn_swallow,//processing data for unwanted write buffers/**< swallowing unnecessary bytes w/o storing */
Conn_closing,//close connection/**< Closing this connection */
Conn_mwrite,//sequential write to multiple item data/**< writing out many items sequentially */
conn_closed,//connection closed/**< connection is closed */
Conn_max_state//MAX State, asserted using/**< max, value (used for assertion) */
};

Next look at the general picture of Drive_machine, which is primarily a while loop to handle the business logic of each state:

The core part of the listener socket and connection socket event callback function:
Finite state machine: performs the corresponding operation according to the state conn_sattes of the socket
static void Drive_machine (conn *c) {
bool stop = FALSE;
int sfd;
Socklen_t Addrlen;
struct Sockaddr_storage addr;
int nreqs = settings.reqs_per_event;
int res;
const char *STR;

ASSERT (c! = NULL);
Because there are conversions or jumps between states, you need to loop until you determine stop
while (!stop) {

corresponding business processing for various states of sockets
Switch (c->state) {
Case conn_listening://Monitoring Status
Addrlen = sizeof (addr);

//
//
//
The main thread enters the state machine and executes the accept operation, which is also non-blocking.
SFD = Accept (C-&GT;SFD, (struct sockaddr *) &addr, &addrlen);
#endif
Connection failed
if (sfd = =-1) {
//
//

}
If the connection is successful, set the connection socket to non-blocking
if (!USE_ACCEPT4) {
if (Fcntl (SFD, F_SETFL, Fcntl (SFD, F_GETFL) | O_nonblock) < 0) {
Perror ("Setting o_nonblock");
Close (SFD);
Break
}
}

If the maximum number of connections is exceeded (as judged by the record of the global State structure), the connection needs to be closed
if (Settings.maxconns_fast &&
Stats.curr_conns + Stats.reserved_fds >= settings.maxconns-1) {
//
//
} else {//If there is no overload, direct distribution (UDP, do not need to establish a connection, direct distribution) worker threads
Dispatch_conn_new (SFD, conn_new_cmd, Ev_read | Ev_persist,
Data_buffer_size, Tcp_transport);
}

Stop = true;
Break

Case conn_waiting:

Case Conn_read:

Case Conn_parse_cmd:

Case Conn_nread:
and various other states

Return
}
}

This section focuses on the first state conn_listening:

The state is the main thread that listens to the socket for business processing: listens for sockets, accepts, and sends the resulting connection socket to a selected worker thread.

Switch (c->state) {
Case conn_listening://Monitoring Status
Addrlen = sizeof (addr);
#ifdef HAVE_ACCEPT4
if (USE_ACCEPT4) {
SFD = Accept4 (C-&GT;SFD, (struct sockaddr *) &addr, &addrlen, Sock_nonblock);
} else {
SFD = Accept (C-&GT;SFD, (struct sockaddr *) &addr, &addrlen);
}
#else
The main thread enters the state machine and executes the accept operation, which is also non-blocking.
SFD = Accept (C-&GT;SFD, (struct sockaddr *) &addr, &addrlen);
#endif
if (sfd = =-1) {
if (use_accept4 && errno = = Enosys) {
USE_ACCEPT4 = 0;
Continue
}
Perror (Use_accept4? "Accept4 ()": "Accept ()");
if (errno = = Eagain | | errno = = ewouldblock) {
/* These is transient, so don ' t log anything */
Stop = true;
} else if (errno = = emfile) {//connection overloaded
if (Settings.verbose > 0)
fprintf (stderr, "Too many open connections\n");
Accept_new_conns (FALSE);
Stop = true;
} else {
Perror ("Accept ()");
Stop = true;
}
Break
}
If the connection is successful, set the connection socket to non-blocking
if (!USE_ACCEPT4) {
if (Fcntl (SFD, F_SETFL, Fcntl (SFD, F_GETFL) | O_nonblock) < 0) {
Perror ("Setting o_nonblock");
Close (SFD);
Break
}
}

If you exceed the number of simultaneous online maximum connections (default is 1024) (judged by the record of the global State structure), you need to close the connection
if (Settings.maxconns_fast &&
Stats.curr_conns + Stats.reserved_fds >= settings.maxconns-1) {
str = "ERROR Too many open connections\r\n";
res = write (sfd, str, strlen (str));
Close (SFD);
Stats_lock ();
stats.rejected_conns++;
Stats_unlock ();
} else {//If there is no overload, direct distribution (UDP, do not need to establish a connection, direct distribution) worker threads
Dispatch_conn_new (SFD, conn_new_cmd, Ev_read | Ev_persist,
Data_buffer_size, Tcp_transport);
}

Stop = true;
Break
}

The choice of the worker thread is in polling (round-robin) mode. Connect the socket to the distribution letter?? is dispath_conn_new:

The main thread is in the callback function of the listener socket, and when a new connection arrives, the function is called to distribute the new connection socket to the worker thread
Note: Because UDP does not need to establish a connection, it is distributed directly to the worker thread
void dispatch_conn_new (int sfd, enum conn_states init_state, int event_flags,
int read_buffer_size, enum Network_transport transport) {
Cq_item *item = Cqi_new ();//Get a free ITEM from the Cq_item resource pool
Char buf[1];
if (item = = NULL) {
Close (SFD);
/* Given that malloc failed this could also fail, but let ' s try */
fprintf (stderr, "Failed to allocate memory for connection object\n");
return;
}

int tid = (last_thread + 1)% settings.num_threads;//Select a thread by Round-robin algorithm

Libevent_thread *thread = threads + tid;//Cache This selected thread

Last_thread = tid;//Update the last selected thread number

Set the fields of the Cq_item
ITEM->SFD = SFD;//SFD is the connection socket
Item->init_state = init_state;
Item->event_flags = Event_flags;
Item->read_buffer_size = read_buffer_size;
Item->transport = transport;

The main thread posts item to the selected worker thread in the item connection queue
Cq_push (Thread->new_conn_queue, item);

Memcached_conn_dispatch (SFD, thread->thread_id);
Buf[0] = ' C ';
Pipeline notification: Writes the character C in the notify_send_fd of the worker thread, indicating that there is a connection
if (Write (THREAD->NOTIFY_SEND_FD, buf, 1)! = 1) {
Perror ("Writing to Thread notify Pipe");
}
}

As you can see, in this distribution function first extracts an item from the Cq_item resource pool (the idle list), sets the information for each field of the connection socket, then selects a worker thread by polling, and then puts the item in the Connection task queue CQ for that worker.    Finally, the notification message is written by notifying the write end of the pipeline. Next is the work thread that has been analyzed earlier to handle all the business of connecting the socket.

Socket connection and dispatch of distributed cache system Memcached State machine