memcached Master-worker Model Analysis __ Network programming

Memcached, I believe we do Linux back-end migrant workers know. Here is a simple analysis of how memcached handles a large number of concurrent connections.

As the title, Memcached is a single process program, single process multi-threaded program (Linuxer may smile, this is not a lot of process). Memcached the bottom is libevent to manage the event, let's take a look at how this libevent's classic application works. In fact, the beginning of the memcached is a genuine single process program, in fact, the use of asynchronous technology can basically put the CPU and the performance of the network card to the limit (this case is simply multithreading will make the program performance drop), but later with the popularity of multi-core CPUs, in order to squeeze the CPU performance, The introduction of multithreading is also homeopathy.

Memcached's source structure is very simple, where thread-related code is basically in THREAD.C. Simply put, Memcached's many threads are a master-worker model, where the main thread is responsible for receiving the connection, then splitting the connection to each worker thread, completing the command reception, processing, and returning the results in each worker thread.

OK, let's start with the main function, step-by-step.

In the main function, thread-related code is basically the following lines:

Case ' t ':
    //Here handle-t parameter, set number of threads.
    //Note below warning, the number of threads exceeds the number of CPU cores in fact, there is no meaning, only negative action
    settings.num_threads = atoi (optarg);
    if (settings.num_threads <= 0) {
        fprintf (stderr, "Number of threads must be greater than");
        return1;
    }
    /* There ' re other problems to above threads.
        * In the future we should portably detect # of cores for the
        * default.
        */
    if (Settings.num_threads >) {
        fprintf (stderr, "warning:setting a high number of worker"
                        "threads Is isn't recommended.\n "" Set this value to the number of
                        cores in "
                        your machine or less.\n");
    break;
 
The thread initialization function is called here, Main_base is the libevent handle of the main thread,
//Because Libevent does not support multithreaded shared handles, each thread has a libevent handle/
* start up worker Threads if MT mode *
/Thread_init (settings.num_threads, main_base);

Below, enter the thread initialization link, before looking at the function of Thread_init, look at a few structural bodies:

Worker thread Structure
typedef struct{
    pthread_t thread_id;        /* Thread ID *
    /structevent_base *base;    * * The libevent handle of this thread
    /structevent notify_event;  /* Notification event, the main thread through this event to notify the worker threads have a new connection
    /intnotify_receive_fd;     /* Notification event associated read FD, this and the following notify_send_fd is a pair of pipes, the specific use behind the said * * *
    intnotify_send_fd;        /* Notification event associated with the write FD, after said * * *
    structthread_stats stats;  /* thread-related STATISTICS believe * * *
    structconn_queue *new_conn_queue/* By the main thread assigned to have not had time to process the connection (client) queue * * *
    cache_t *suffix_cache;      /* Suffix cache *
/} libevent_thread;
 
This is the main thread of the structure, it is relatively simple
//The example of this structure is only a global dispatcher_thread
typedef struct{
    pthread_t thread_id;        /* Main thread ID *
    /structevent_base *base;    /* LIBEVENT handle *
/} libevent_dispatcher_thread;

The following goes into the thread initialization function:

void Thread_init (int nthreads,struct event_base *main_base) {int i; 
 
    First initialize a heap of locks//This is the primary lock, used to synchronize Key-value cache access Pthread_mutex_init (&cache_lock, NULL);
 
    This is the cache state lock, which is used to synchronize the access of some statistic data of memcached pthread_mutex_init (&stats_lock, NULL);
 
    This lock is used to synchronize the access Pthread_mutex_init (&init_lock, NULL) of the variable Init_count (number of initialized threads);
 
    This is the condition variable pthread_cond_init (&init_cond, NULL) that notifies all threads to initialize the completion;
    This lock is used to synchronize the access Pthread_mutex_init (&cqi_freelist_lock, NULL) of the free connection list;
 
    Cqi_freelist = NULL;
    Allocate worker thread structure body Deposit threads = calloc (nthreads,sizeof (Libevent_thread));
        if (! Threads) {perror ("Can ' t allocate thread descriptors");
    Exit (1);
    //Set the main thread first dispatcher_thread.base = Main_base;
 
    dispatcher_thread.thread_id = Pthread_self ();
        Sets the pipe between all worker threads and the main thread for (i = 0; i < nthreads; i++) {intfds[2];
            if (pipe (FDS)) {perror ("Can ' t create notify Pipe");
   Exit (1);     } THREADS[I].NOTIFY_RECEIVE_FD = Fds[0];
 
        THREADS[I].NOTIFY_SEND_FD = fds[1];
    This function carries out the initialization of the worker thread, such as libevent handle, Connection queue initialization setup_thread (&threads[i]); //This is where the real call Pthread_create created the thread for (i = 0; i < nthreads; i++) {Create_worker (worker_libevent, &am
    P;threads[i]);
    }//main thread all the worker threads are running up and running back after the code (accept connection) Pthread_mutex_lock (&init_lock);
    while (Init_count < nthreads) {pthread_cond_wait (&init_cond, &init_lock);
} pthread_mutex_unlock (&init_lock); }

OK, initialization is complete, each thread (including the main thread) is running, so let's look at how the specific connection is handled.

First look at the main thread, after the Thread_init return (all thread initialization complete), the main function did some other initialization after the call to Event_base_loop (main_base, 0); This function begins to handle network events and accepts connections. Prior to this, the main function, when binding the listening port, had already added the listening socket event to the Main_base (see the Server_socket function, not much). The callback function that listens for an event is a callback function that is common to all network events in memcached Event_handler, and this event_handler is basically doing nothing, directly calling Drive_machine, This function is made up of a large state machine that is a big switch. Here is the memcached of all network events, let's take a look at:

static void Drive_machine (conn *c) {boolstop = false;
    INTSFD, flags = 1;
    Socklen_t Addrlen;
    Structsockaddr_storage addr;
    Intnreqs = settings.reqs_per_event;
 
    Intres;
            while (!stop) {switch (c->state) {//This monitoring state only the main thread of the listening FD will have, and the main thread is basically such a State caseconn_listening:
            Here, there is a new connection to the//accept new connection Addrlen = sizeof (addr); if ((sfd = Accept (C->SFD, (STRUCTSOCKADDR *) &addr, &addrlen)) = = 1) {if (errno== Eagain | | errn
                o== ewouldblock) {/* are transient, so don ' t log anything/stop = true; }elseif (errno== emfile) {if (Settings.verbose > 0) fprintf (s
                    Tderr, "Too many open connections\n");
                    Accept_new_conns (FALSE);
                Stop = true;
                    }else{perror ("Accept ()");
       Stop = true;         } break;
                //Set Socket non-blocking if (flags = Fcntl (SFD, F_GETFL, 0) < 0 | | Fcntl (SFD, F_SETFL, Flags |
                O_nonblock) < 0) {perror ("setting o_nonblock");
                Close (SFD);
            Break ///Assign the new connection to the worker thread Dispatch_conn_new (SFD, conn_new_cmd, Ev_read |
            Ev_persist, Data_buffer_size, Tcp_transport);
            Stop = true;
 
            Break
 
    The following are some of the events of the worker thread, which are slightly caseconn_waiting://... Caseconn_read://...}
Return }

Then look at the Dispatch_conn_new function:

void dispatch_conn_new (int sfd,enum conn_states init_state, int event_flags, int r Ead_buffer_size,enum Network_transport Transport) {//Assign a connection queue item, this item will be plugged into the worker thread's connection queue by the main thread cq_item *item = CQI
 
    _new ();
    RR polling gets the target thread for this connection Inttid = (last_thread + 1)% Settings.num_threads;
    Libevent_thread *thread= threads + tid;
 
    Last_thread = tid;
    Initialize Item ITEM->SFD = SFD;
    Item->init_state = init_state;
    Item->event_flags = Event_flags;
    Item->read_buffer_size = read_buffer_size;
 
    Item->transport = transport;
 
    Plug the item into the worker thread's queue Cq_push (Thread->new_conn_queue, item);
    Memcached_conn_dispatch (sfd,thread->thread_id); Writes a byte to the notification of a worker thread, so that NOTIFY_RECEIVE_FD will have a byte to read//So the notify_event of the worker thread will receive a readable event/  Memcached is this way to achieve the purpose of asynchronous notification between threads, very tricky if (THREAD->NOTIFY_SEND_FD, "", 1)!= 1) {perror ("writing to Thread
    Notify Pipe "); }
}

Well, since the logic of this main thread processing connection is basically gone, here's a look at the code for the worker thread. After the worker thread initialization completes, the callback for the Notify_event libevent event is registered to the thread_libevent_process, to see:

static void thread_libevent_process (int fd,short which,void *arg) {libevent_thread *me = arg;
    Cq_item *item;
 
    CHARBUF[1]; Reads a byte written by the main thread, and a byte represents a connection if (read (FD, buf, 1)!= 1) if (Settings.verbose > 0) fprintf (stderr, "C
 
    An ' t-read from libevent pipe\n ');
 
    Connect the main thread to the queue pop out item = Cq_pop (Me->new_conn_queue); if (NULL!= Item) {///Initialize new connection, register event listener, callback to previous mentioned Event_handler conn *c = Conn_new (ITEM->SFD, item->init_s
        Tate, Item->event_flags, Item->read_buffer_size, Item->transport, me->base);  if (c = = NULL) {if (IS_UDP (Item->transport)) {fprintf (stderr, "Can" T listen for events on
                UDP socket\n ");
            Exit (1); }else{if (Settings.verbose > 0) {fprintf (stderr, "Can" T listen for the events on FD%d\
                n ", ITEM->SFD); Close (item->SFD);
        }}else{c->thread= me;
    ///Recycle item Cqi_free (item); }
}

All right, so the worker thread has one more connection, and the worker thread behind it is the network IO event processing business that constantly listens for notify events to add connections and client connections to sockets.

Memcached's multithreaded libevent mechanism is almost a textbook for high-performance servers. Linux back-end migrant workers must read.