Distributed cache system Memcached main thread main function

In the first two sections, the workflow of work threads is analyzed in more detail, and the main process is summarized as follows:

Next this section mainly analyzes the main thread related function design, main function main the basic flow as shown:

For the initialization of a worker thread in the main thread before all worker threads have been analyzed, the subsequent creation of a listening socket, registering the Libevent event to monitor the socket, and starting the Libevent event loop of the main thread, is what follows.

The function that is primarily called is server_sockets, which extracts an IP from the configuration parameter Setting.inner string, or a hostname (a hostname may have multiple IPs), and then passes it to the function Server_ The socket function handles it. The Server_socket function is responsible for completing the socket creation, binding to the port, listening to the socket, and the conn structure of the listener socket (calling function conn_new, in which the listener socket is registered to the main thread libevent main_ On base, the callback function is Event_handler, the core part of which is drive_machine, which is consistent with the worker thread, and then puts the conn into the listening queue (conn *listen_conn). The conn of the client connection received on the listener will be placed in the connection queue Conn **conns. Finally, start the Libevent event loop in the main function. Or is the diagram intuitive, as follows:

static int server_sockets (int port, enum Network_transport Transport,
FILE *portnumber_file) {//<span style= "Color:rgb (0, 0); Font-family:consolas, ' Courier New ', Courier, Mono, se Rif font-size:11.8518514633179px; line-height:18px; " >port is the default 11211 or the port number set by the user using the-P option </span>

There may be more than one IP address in the Settings.inter. If there are more than one then comma separated
Char *b;
int ret = 0;
Copy a string to avoid the following Strtok_r function modification (pollution) global variable Settings.inter
Char *list = StrDup (Settings.inter);

This cycle is primarily about handling multiple IPs
for (char *p = strtok_r (list, ";,", &b);
P! = NULL; Separate the IP, using semicolons; as separators
p = strtok_r (NULL, ";,", &b)) {
int the_port = port;
Char *s = STRCHR (P, ': ');//startup may use the-l ip:port Parameter form
IP followed by the port number, that is, the specified IP and also specify the port number of the IP
This takes the port number behind the IP instead of the port number specified by-p
if (s! = NULL) {
*s = ' + ';//truncate the trailing port number so that the string that P points to is just an IP
++s;
if (!safe_strtol (S, &the_port)) {//illegal port number parameter value
return 1;
}
}
if (strcmp (P, "*") = = 0) {
p = NULL;
}
Handle one of the IPs. have p specified IP (or hostname)
RET |= server_socket (p, The_port, Transport, portnumber_file);
}
Free (list);
return ret;
}


Static conn *listen_conn = null;//Listener queue (may be listening to multiple IPs at the same time)

Interface is an IP, hostname, or null. There is no port number behind this IP string. Port number is indicated by the parameter port
static int server_socket (const char *interface,
int port,
Enum Network_transport Transport,
FILE *portnumber_file) {
int sfd;
struct Linger Ling = {0, 0};
struct Addrinfo *ai;
struct Addrinfo *next;
struct Addrinfo hints = {. Ai_flags = ai_passive,
. ai_family = Af_unspec};
Char Port_buf[ni_maxserv];
int success = 0;
int flags = 1;

Hints.ai_socktype = IS_UDP (transport)? Sock_dgram:sock_stream;


snprintf (port_buf, sizeof (PORT_BUF), "%d", port);
Getaddrinfo (interface, Port_buf, &hints, &ai);

If interface is a hostname, then there may be multiple IPs
for (next= Ai; next; next= Next->ai_next) {
Conn *listen_conn_add;

Create a socket and set it to a non-blocking
SFD = New_socket (next);//Calling the socket function
Bind (SFD, next->ai_addr, Next->ai_addrlen);

success++;
Listen (sfd, settings.backlog);

The function conn_new registers the listener socket FD on the main_base and sets the callback function to Event_handler, where the core is the Drive_machine function, which is consistent with the worker thread
if (! ( Listen_conn_add = Conn_new (SFD, conn_listening,
Ev_read | Ev_persist, 1,
Transport, Main_base))) {
fprintf (stderr, "failed to create listening connection\n");
Exit (Exit_failure);
}

Put multiple conn that will be listening to a listening queue
Listen_conn_add->next = Listen_conn;
Listen_conn = Listen_conn_add;

}

Freeaddrinfo (AI);

/* Return Zero IFF we detected no errors in starting up Connections */
return success = = 0;
}


static int new_socket (struct addrinfo *ai) {
int sfd;
int flags;
SFD = socket (ai->ai_family, Ai->ai_socktype, Ai->ai_protocol);
Flags = Fcntl (SFD, F_GETFL, 0);
Fcntl (SFD, F_SETFL, Flags | O_nonblock);

return SFD;
}

The main thread for each listener socket and each received client connection socket is assigned a conn structure to manage the various state information of the socket. It is important to note that memcached does not create a single conn structure for each socket, but rather a pointer to a conn structure at the time of initialization (which is related to the maximum number of client connections approved at the same time) (note that it is not a conn struct, Because each conn structure is relatively large, so if the direct allocation of a number of conn structures need to occupy a larger space, when you do need a conn structure, and then from the pre-allocated pointer array to take one, and actually allocate space for the pointer, complete the specific initialization and so on. This is consistent with the previous Cq_item memory pool-pre-allocated by configuration, available on demand, and recycled. Avoid memory fragmentation and improve performance.

Where the function conn_init is responsible for the pre-allocation set of several conn pointers, maintained by a conn** pointer. The function conn_new obtains a conn* from the conn** maintained array when it does need a conn, and completes the actual space allocation.

The specific analysis is as follows:

function Conn_init:

Conn **conns; Conn Array Pointers
static void Conn_init (void) {
/* We ' re unlikely to see an FD much higher than Maxconns. */
The DUP has returned the currently unused minimum positive integer, so next_fd equals the number of FD that has been consumed at this moment
int next_fd = DUP (1);//Gets the number of FD currently in use
Some file descriptors are reserved. That is, more applications for some conn structures. To avoid the need to put the file descriptor
to the account. Causes the value of the socket FD to be greater than this array length
int headroom = 10;//reserved Some file descriptors/* account for extra unexpected open FDs */
struct Rlimit RL;

The default value for Settings.maxconns is 1024.
Max_fds = Settings.maxconns + headroom + next_fd;

/* If possible, get the actual highest FD we can possibly ever see. */
if (Getrlimit (rlimit_nofile, &rl) = = 0) {
Max_fds = Rl.rlim_max;
} else {
fprintf (stderr, "Failed to query maximum file descriptor;
"Falling Back to maxconns\n");
}

Close (NEXT_FD);//NEXT_FD is only used for counting, and there is no other use

Note that the number of Conn structures requested is more than the number of Settings.maxconns this client simultaneously online
It's bigger. Because memcached is directly using the value of the socket FD as an array subscript. It is
This one?? Because, you need to use headroom to reserve some space for unexpected situations.
if (Conns = calloc (Max_fds, sizeof (conn *))) = = NULL) {//Note that the conn pointer is not a conn struct
fprintf (stderr, "Failed to allocate Connection structures\n");
/* This is unrecoverable so bail out early. */
Exit (1);
}
}

function Conn_new:

Assign a conn structure to the SFD, and create an event for the SFD and register it on the event_base.
Conn *conn_new (const int SFD, enum conn_states init_state,//init_state value conn_listening
const int Event_flags,
const int read_buffer_size, enum Network_transport Transport,
struct Event_base *base) {
Conn *c;

ASSERT (SFD >= 0 && sfd < Max_fds);
c = conns[sfd];//Direct use subscript

if (NULL = = c) {//No previous connection used this SFD value, need to apply for a conn struct
if (! ( c = (conn *) calloc (1, sizeof (conn)))) {
fprintf (stderr, "Failed to allocate Connection object\n");
return NULL;
}

...//Initialize some member variables

C-&GT;SFD = SFD;
CONNS[SFD] = c; Put this struct to Conns array management
}

...//Initialize additional member variables
C->state = init_state;//value is conn_listening

Equivalent to Event_assign, the current_base is automatically associated. The callback function of the event is Event_handler
Event_set (&c->event, SFD, Event_flags, Event_handler, (void *) c);
Event_base_set (base, &c->event);
C->ev_flags = Event_flags;

if (Event_add (&c->event, 0) = =-1) {
Perror ("Event_add");
return NULL;
}

return C;
}

As can be seen from the above, virtually all conn from pre-allocation to actual allocation are maintained by conn** pointers. It is only necessary to determine whether an element in the array conn the pointer is empty: equal to NULL is not actually occupied, is idle, and vice versa, has been occupied by a real socket FD.

At this point, all the preparation for the main thread is ready, and then the real client connection event is processed:

The callback function Event_handler (which is also the callback function of the worker thread's registration event):

The Event_handler itself is simple, and its core is the Drive_machine function (a finite state machine that handles all customer logic).

void Event_handler (const int FD, const short which, void *arg) {
Conn *c;

c = (conn *) arg;
ASSERT (c! = NULL);

C->which = which;
if (fd! = C->SFD) {
Conn_close (c);
Return
}

Drive_machine (c);
Return
}

The drive_machine is still relatively complex, and next will be divided into several sections of the content, on this thin way.

Distributed cache system Memcached main thread main function