Memcached source code analysis ----- LRU queue and item struct, memcached ----- lru

Memcached source code analysis ----- LRU queue and item struct, memcached ----- lru

Reprinted please indicate the source: http://blog.csdn.net/luotuo44/article/details/42869325

LRU queue:

In the previous slab Memory Allocation blog, all slab splitters in an slab class are allocated with the same size of items, and different slab classes are allocated with different sizes of items. The item struct contains a slabs_clsid member to indicate which slab class it belongs. Here, the items with the same slabs_clsid value are called the same class items.

The slab distributor is responsible for allocating an item, but this item is not managed directly for the hash table. You can see from "delete a hash table" that deleting an item in a hash table simply deletes this item from the conflict chain of the hash table, the item memory is not returned to slab. In fact, the items allocated by the slab distributor are managed by an LRU queue. When an item is deleted from the LRU queue, it is returned to the slab distributor.

The LRU queue is actually a two-way linked list. There are multiple LRU queues in memcached, and the number is equal to the number of types of items. Therefore, the same type of item (slabs_clsid member variable values are equal) will be placed in the same LRU queue. It is a two-way linked list, because it is convenient to insert a table from both the front and back directions and to traverse the chain table. The following describes some members of the item struct.

Typedef struct _ stritem {struct _ stritem * next; // next pointer, used for the LRU chain table struct _ stritem * prev; // prev pointer, used for the LRU chain table struct _ stritem * h_next; // h_next pointer, used to hash the table's conflict chain rel_time_t time; // The last access time. Absolute Time... uint8_t slabs_clsid;/* which slab class we're in */} item;

Next, let's take a look at the data structure for managing the LRU queue. It's actually super simple. It's just three global arrays, and it's a static array (happy ).

// Memcached. hfile # define POWER_LARGEST 200 // items. file c # define LARGEST_ID POWER_LARGESTstatic item * heads [LARGEST_ID]; // point to the static item * tails [LARGEST_ID] of each LRU queue header; // point to the end of each LRU queue static unsigned int sizes [LARGEST_ID]; // number of items in each LRU queue

The size of the three arrays is the same as the maximum number of slab classes provided by slabclass. This also confirms that an LRU queue corresponds to a class of items. Heads [I] points to the first item of the I-class LRU linked list, and tails [I] points to the last item of the I-class LRU linked list, sizes [I] indicates the number of items in the I-class LRU linked list. Shows the structure composed of LRU queue and heads and tails:

Basic LRU queue operations:

Insert operation:

Suppose we want to insert an item into the LRU linked list, we can call the item_link_q function to insert the item into the LRU queue. The following is the specific implementation code.

// Insert the item into the head of the LRU queue static void item_link_q (item * it) {/* item is the new head */item ** head, ** tail; assert (it-> slabs_clsid <LARGEST_ID); assert (it-> it_flags & ITEM_SLABBED) = 0); head = & heads [it-> slabs_clsid]; tail = & tails [it-> slabs_clsid]; assert (it! = * Head); assert (* head & * tail) | (* head = 0 & * tail = 0 )); // Insert the item it-> prev = 0; it-> next = * head; if (it-> next) it-> next-> prev = it; * head = it; // This item serves as the first node of the corresponding linked list. // if this item is the first item on the corresponding id, it will also be considered as the last item on the id chain. // because headers are used in the head, the first item to be inserted, the last item if (* tail = 0) * tail = it; sizes [it-> slabs_clsid] ++; // The number of items plus a return ;}

Delete operation:

With the insert function, there must be a corresponding delete function. The delete function is quite simple. It mainly deals with how the pre-and post-drive nodes of the node are merged after the node is deleted.

// Delete it from the corresponding LRU queue static void item_unlink_q (item * it) {item ** head, ** tail; assert (it-> slabs_clsid <LARGEST_ID ); head = & heads [it-> slabs_clsid]; tail = & tails [it-> slabs_clsid]; if (* head = it) {// assert (it-> prev = 0); * head = it-> next;} if (* tail = it) {// assert (it-> next = 0); * tail = it-> prev;} assert (it-> next! = It); assert (it-> prev! = It); // connect the item's front node to the rear drive node if (it-> next) it-> next-> prev = it-> prev; if (it-> prev) it-> prev-> next = it-> next; sizes [it-> slabs_clsid] --; // The number of digits minus one return ;}

It can be seen that whether an item is inserted or deleted, the time consumed is a constant. In fact, for memcached, the time complexity of almost all operations is constant.

Update operation:

Why insert an item to the head of the LRU queue? Of course, one of the reasons is that the implementation is simple. But more importantly, this is an LRU queue !! Remember the LRU in the operating system. This is an elimination mechanism. In the LRU queue, the lower the backend is considered to be less-used items, the higher the chances of being eliminated. Therefore, fresh items (new access time) must be placed in front of less fresh items, so the header inserted into the LRU queue is the best choice. The following do_item_update function proves this. The do_item_update function first deletes the old item from the LRU queue, and then inserts it into the LRU Queue (at this time, it ranks first in the LRU Queue ). In addition to updating the position of an item in the queue, it also updates the time member of the item, which specifies the last access time (absolute time ). If it is not for LRU, the simplest implementation of the do_item_update function is to directly update the time member.

# Define ITEM_UPDATE_INTERVAL 60 // The update frequency is 60 seconds. void do_item_update (item * it) {// The following Code shows that the update operation is time-consuming. If this item is frequently accessed, // will lead to excessive update and a series of time-consuming operations. The Update Interval comes into being. If the last access time (or the update time) is within the current (current_time) // still updating interval, it will not be updated. Update only when the quota is exceeded. If (it-> time <current_time-ITEM_UPDATE_INTERVAL) {mutex_lock (& cache_lock); if (it-> it_flags & ITEM_LINKED )! = 0) {item_unlink_q (it); // delete it from the LRU queue-> time = current_time; // update the access time item_link_q (it ); // Insert the header to the LRU queue} mutex_unlock (& cache_lock );}}

When memcached processes the get command, it calls the do_item_update function to update the item access time and its location in the LRU queue. In memcached, the get command is a frequently used command. The items that rank first or first in the LRU queue are frequently get. For the first few items, calling do_item_update is of little significance, because the location after calling do_item_update is still the first few items, in addition, it is difficult to eliminate multiple items after LRU is eliminated (there are many items in an LRU Queue ). On the other hand, the do_item_update function consumes a certain amount of time because it needs to seize the cache_lock lock. If the do_item_update function is frequently called, the performance will decrease a lot. Therefore, memcached uses the update interval.

I have discussed how to insert and delete an item in the LRU queue. Now I will explain how to apply for an item from the slab distributor and how to return the item to the slab distributor.

Item struct:

Although we have mentioned item many times before, it is estimated that the reader is confused about what it looks like. Next, let's take a look at the complete definition of the item struct. Pay attention to the English comments.

# Define ITEM_LINKED 1 // This item is inserted into the LRU queue # define ITEM_CAS 2 // This item uses CAS # define ITEM_SLABBED 4 // This item is still in the slab idle queue, not allocated # define ITEM_FETCHED 8 // after this item is inserted into the LRU queue, the worker thread has accessed typedef struct _ stritem {struct _ stritem * next; // next pointer, used for LRU chain table struct _ stritem * prev; // prev pointer, used for LRU chain table struct _ stritem * h_next; // h_next pointer, used for hash table conflict chain rel_time_t time; // The last access time. Absolute Time rel_time_t exptime; // expiration time, absolute time int nbytes; // The length of the data stored in this item unsigned short refcount; // number of references of this item uint8_t nsuffix; // suffix length/* length of flags-and-length string */uint8_t it_flags; // item attributes/* ITEM _ * above */uint8_t slabs_clsid; /* which slab class we're in */uint8_t nkey; // key value length/* key length, w/terminating null and padding * // * this odd type prevents type-punning issues when we do * the little Shuffle to save space when not using CAS. */union {uint64_t cas; char end;} data []; /* if it_flags & ITEM_CAS we have 8 bytes CAS * // * then null-terminated key * // * then "flags length \ r \ n" (no terminating null) * // * then data with terminating \ r \ n (no terminating null; it's binary !) */} Item;

The English comment in the above Code illustrates the item layout. The last member of the item struct is data []. Such a definition is called a flexible array. One feature of flexible arrays is that data fields are stored behind arrays. The items of memcached also use flexible arrays. The item above only defines the members of the item struct, but the previous blog post always uses the item to represent the data stored by the item. This is reasonable. Because the item struct itself and the data corresponding to the item are stored in the same memory allocated by the slab distributor. When the slabclass array is initialized in slab Memory distributor, the size of the allocated memory block is sizeof (item) + settings. chunk_size.

The item struct is followed by a pile of data, not just the data to be stored by the user, as shown in:

The figure above may not completely understand the storage format of items and corresponding data. I understand that coders need code to be completely clear.

# Define ITEM_key (item) (char *) & (item)-> data) \ + (item)-> it_flags & ITEM_CAS )? Sizeof (uint64_t): 0) # define ITEM_suffix (item) (char *) & (item)-> data) + (item) -> nkey + 1 \ + (item)-> it_flags & ITEM_CAS )? Sizeof (uint64_t): 0) # define ITEM_data (item) (char *) & (item)-> data) + (item) -> nkey + 1 \ + (item)-> nsuffix \ + (item)-> it_flags & ITEM_CAS )? Sizeof (uint64_t): 0) # define ITEM_ntotal (item) (sizeof (struct _ stritem) + (item)-> nkey + 1 \ + (item) -> nsuffix + (item)-> nbytes \ + (item)-> it_flags & ITEM_CAS )? Sizeof (uint64_t): 0) static size_t item_make_header (const uint8_t nkey, const int flags, const int nbytes, char * suffix, uint8_t * nsuffix) {/* suffix is defined at 40 chars elsewhere .. */* nsuffix = (uint8_t) snprintf (suffix, 40, "% d \ r \ n", flags, nbytes-2); return sizeof (item) + nkey + * nsuffix + nbytes; // calculate the total size} // The key, flags, and exptime parameters are input when you use the set and add commands to store a piece of data. // Nkey is the length of the key string. Nbytes is the data length to be stored by the user + 2, because "\ r \ n" is added at the end of data, it is the hash value calculated based on the key value. Item * keys (char * key, const size_t nkey, const int flags, const rel_time_t exptime, const int nbytes, const uint32_t cur_hv) {uint8_t nsuffix; item * it = NULL; char suffix [40]; // total space required to store this item. Note that the first parameter is nkey + 1, so the above macro computing // used (item)-> nkey + 1 size_t ntotal = item_make_header (nkey + 1, flags, nbytes, suffix, & nsuffix); if (settings. use_cas) {// enable the CAS function ntotal + = sizeof (uint64_t);} // determine the slab unsigned int id = slabs_clsid (ntotal) based on the size ); if (id = 0) // 0 indicates that it does not belong to any slab return 0 ;... it = slabs_alloc (ntotal, id); // apply for memory it from the slab distributor-> refcount = 1; it-> it_flags = settings. use_cas? ITEM_CAS: 0; it-> nkey = nkey; it-> nbytes = nbytes; memcpy (ITEM_key (it), key, nkey); // only nkey bytes are copied here, the last byte is empty with it-> exptime = exptime; memcpy (ITEM_suffix (it), suffix, (size_t) nsuffix); it-> nsuffix = nsuffix; return it ;}

Coders should have a good understanding of the macro definitions in the code above and the item_make_header function. The code above is a simple introduction to the do_item_alloc function. This function is actually quite complex.

 

The above is a simple example of applying for an item from slab, and now the code for returning the item is pasted.

// Items. file c void item_free (item * it) {size_t ntotal = ITEM_ntotal (it); unsigned int clsid; clsid = it-> slabs_clsid; it-> slabs_clsid = 0; slabs_free (it, ntotal, clsid);} // slabs. file c void slabs_free (void * ptr, size_t size, unsigned int id) {pthread_mutex_lock (& slabs_lock); do_slabs_free (ptr, size, id ); // return it to slab distributor pthread_mutex_unlock (& slabs_lock );}

Contact between item and hash table:

The previous do_item_alloc function applies for an item based on the required size. According to the do_item_alloc implementation code, it does not insert this item into the hash table and LRU queue. In fact, this task is implemented by another function.

Next, let's take a look at how to pass the item to the hash table, LRU queue, and how to retrieve the item from the hash table and LRU Queue (sometimes it needs to be returned to slab ).

// Insert the item into the hash table and LRU queue. The hash value hvint do_item_link (item * it, const uint32_t hv) is required to insert the item into the hash table) {// make sure that this item has been allocated from slab and has not been inserted into the LRU queue assert (it-> it_flags & (ITEM_LINKED | ITEM_SLABBED) = 0 ); // when the hash table does not migrate data for expansion, the item is inserted into the hash table. // This lock is occupied when the hash table migrates data. Mutex_lock (& cache_lock); it-> it_flags | = ITEM_LINKED; // Add the link flag it-> time = current_time;/* Allocate a new cas id on link. */ITEM_set_cas (it, (settings. use_cas )? Get_cas_id (): 0); assoc_insert (it, hv); // insert this item into the hash table item_link_q (it ); // insert this item into the linked list refcount_incr (& it-> refcount); // reference count plus mutex_unlock (& cache_lock); return 1 ;}// delete from the hash table, therefore, the hash value hvvoid do_item_unlink (item * it, const uint32_t hv) {mutex_lock (& cache_lock); if (it-> it_flags & ITEM_LINKED )! = 0) {it-> it_flags & = ~ ITEM_LINKED; // subtract the link mark assoc_delete (ITEM_key (it), it-> nkey, hv); // Delete item_unlink_q (it) from the hash table ); // delete this item do_item_remove (it) from the linked list; // return this item to slab} mutex_unlock (& cache_lock);} void do_item_remove (item * it) {assert (it-> it_flags & ITEM_SLABBED) = 0); assert (it-> refcount> 0); if (refcount_decr (& it-> refcount) = 0) {// return item_free (it) when the reference count is equal to 0; // return this item to slab }}

When the do_item_remove function is used to return an item to slab, the system will first test whether the reference number of this item is equal to 0. The reference number can be simply understood as whether the worker thread is using this item. The reference number will be detailed in the following blog.

The lock is rarely seen in the previous code. In fact, the previous item operations require locking, because multiple worker threads may simultaneously operate the hash table and LRU queue. The reason why the lock is rarely seen is that they all use the wrap function (if you have read the UNIX network programming, this concept should not be unfamiliar ). Lock and unlock the Package function. In the preceding functions, the function names are generally prefixed with do. The function name corresponding to the package is to remove the prefix do _. The introduction of the lock is not a task in this blog post. A special blog post will be held to introduce the use of the lock.