In-depth analysis of memcached's thread access model-I

Memcached is a distributed memory cache system. It is widely used in a variety of sites to continuously improve the overall access performance of the site, memcached is very simple to use. It can be said that the use threshold is very low, which may be one of the reasons for memcached's popularity.

 

We can see that there are many articles on analyzing memcached on the Internet. This article is an in-depth analysis of the thread access model based on the source code of memcached, by studying and drawing on some ideas in the design and implementation of memcached, we will add new ideas for our system architecture design.

 

Simply put, the implementation of memcached does not use a multi-process model, but instead selects a multi-threaded model. It can be said that this implementation solution is indeed relatively lightweight. Multithreading is implemented based on the libevent event mechanism and pipeline pipe-based signaling mechanism.

 

After the main thread main_thread initializes the system and generates a specified number of worker_threads, it converts itself into a dispatch_thread, which is responsible for system listening and forwarding. main_thread has its own event_base instance, this instance is responsible for monitoring whether a socket connection has arrived. If yes, it obtains a new connection socket and encapsulates it into a cq_item object into the CQ queue of A worker_thread. At the same time, write a byte to the worker y pipe of the worker_thread to upload the new cq_itme to the worker_thread, which can be processed.

 

Each worker_thread has its own CQ queue, which stores various connection objects that have established connections. worker_thread continuously obtains various connection objects from their respective CQ queues for processing. At the same time, worker_thread also has its own event_base instances used for socket read/write operations and a pair of read/write pipe for signaling workers.

 

Below are simple main_thread and worker_thread:

 

 

(Http://user.qzone.qq.com/270362985? Ptlang = 2052, view the image here)

 

 

 

The following combined with the source code (memcached-1.4.4) for analysis, home page to see the key data structure definition.

 

1. Definitions of connection queues and connection items stored in each worker_thread

 

View plaincopy to clipboardprint?
/*
An item in the connection queue.
Encapsulate the connected socket file description
*/

Typedef struct conn_queue_item cq_item;

Struct conn_queue_item {

Int SFD; // socket file description for establishing a connection

Enum conn_states init_state; // indicates the connection status.

Int event_flags; // the flags of the event

Int read_buffer_size; // read buffer size

Enum network_transport transport; // The transmission protocol corresponding to the connection: TCP or UDP

Cq_item * Next; // pointer to the next cq_item

};
/*
An item in the connection queue.
Encapsulate the connected socket file description
*/

Typedef struct conn_queue_item cq_item;

Struct conn_queue_item {

Int SFD; // socket file description for establishing a connection

Enum conn_states init_state; // indicates the connection status.

Int event_flags; // the flags of the event

Int read_buffer_size; // read buffer size

Enum network_transport transport; // The transmission protocol corresponding to the connection: TCP or UDP

Cq_item * Next; // pointer to the next cq_item

};

 

 

The cq_item instance is the encapsulation of main_thread accept after a socket connection.

 

View plaincopy to clipboardprint?
/* A connection queue .*/

Typedef struct conn_queue CQ;

Struct conn_queue {

Cq_item * head;

Cq_item * tail;

Pthread_mutex_t lock;

Pthread_cond_t cond;

};
/* A connection queue .*/

Typedef struct conn_queue CQ;

Struct conn_queue {

Cq_item * head;

Cq_item * tail;

Pthread_mutex_t lock;

Pthread_cond_t cond;

};

 

 

CQ is the established connection queue that each worker_thread stores. Each worker_thread is constantly processed by reading the cq_item of its own connection queue.

 

2. In fact, memcached encapsulates each worker_thread into a corresponding struct.

To facilitate the processing of worker_thread, memcache re-encapsulates worker_thread. The structure is as follows:

View plaincopy to clipboardprint?
/*
 
* Tenfyguo: memcached encapsulates each thread. Each thread has its own event_base OBJ and corresponding conn_queue queue.
 
*/

Typedef struct {

Pthread_t thread_id;/* unique ID of this thread */

Struct event_base * base;/* libevent handle this thread uses */



Struct event policy_event;/* Listen event for pipeline y Pipe */



Int notify_receive_fd;/* processing ing end of running y Pipe */

Int notify_send_fd;/* sending end of running y Pipe */



Struct thread_stats stats;/* stats generated by this thread */



Struct conn_queue * new_conn_queue;/* queue of new connections to handle */



Cache_t * suffix_cache;/* suffix cache */



} Libevent_thread;
/*

* Tenfyguo: memcached encapsulates each thread. Each thread has its own event_base OBJ and corresponding conn_queue queue.

*/

Typedef struct {

Pthread_t thread_id;/* unique ID of this thread */

Struct event_base * base;/* libevent handle this thread uses */

Struct event policy_event;/* Listen event for pipeline y Pipe */

 

Int notify_receive_fd;/* processing ing end of running y Pipe */

Int notify_send_fd;/* sending end of running y Pipe */

Struct thread_stats stats;/* stats generated by this thread */

Struct conn_queue * new_conn_queue;/* queue of new connections to handle */

 

Cache_t * suffix_cache;/* suffix cache */

} Libevent_thread;
 

 

 

The following is the encapsulation of main_thread as a dispatch_thread:

 

View plaincopy to clipboardprint?
Typedef struct {

Pthread_t thread_id;/* unique ID of this thread */

Struct event_base * base;/* libevent handle this thread uses */

} Libevent_dispatcher_thread;
Typedef struct {

Pthread_t thread_id;/* unique ID of this thread */

Struct event_base * base;/* libevent handle this thread uses */

} Libevent_dispatcher_thread;

 

 

 

 

After introducing the main data structure (of course there are other important data structures, but it is not very closely related to the thread we will introduce in this article, so we will not extend it here ), the following describes the key processes.

 

This article from the csdn blog, reproduced please indicate the source: http://blog.csdn.net/tenfyguo/archive/2010/01/31/5273828.aspx