memcached Source Analysis-----LRU Queue and item structure

Reprint Please specify Source: http://blog.csdn.net/luotuo44/article/details/42869325

LRU Queue:

The previous "Slab Memory allocation" blog post has said that all slab allocators in a slab class are assigned only the same size item, and different slab classes are assigned different sizes of item. The item structure has a SLABS_CLSID member that indicates which slab class it belongs to. here to Slabs_clsid values of the same Item called the same class Item .

The slab allocator is responsible for assigning an item, but this item is not administered directly to the hash table. From the delete operation of the hash table, it can be seen that deleting an item in a hashtable simply removes the item from the conflict chain of the Hashtable and does not return the item memory to slab. In fact, the item assigned by the slab allocator is managed by an LRU queue. When an item is deleted from the LRU queue, it is returned to the slab allocator.

The LRU queue is actually a doubly linked list. There are multiple LRU queues in the memcached, with the number of items equal to the type of item. So, the same class of item (the SLABS_CLSID member variable value is equal) will be placed in the same LRU queue. It is a doubly linked list because it is convenient to insert and traverse the linked list from the front and back two directions. Let's look at some of the members of the item structure.

typedef struct _STRITEM {    struct _stritem *next;//next pointer for LRU linked list    struct _stritem *prev;//prev pointer for LRU linked list    struct _stritem *h_next;//h_next pointer, used for hash table conflict chain    rel_time_t time      ;   Last access time. Absolute time  ...    uint8_t         slabs_clsid;/* which slab class we ' re in * *} item;

Next, look at the data structure that manages the LRU queue. In fact, it is super simple, just three global arrays, but also a static array (happy bar).

Memcached.h File # define Power_largest  200//items.c file # define LARGEST_ID power_largeststatic Item *heads[largest_ id];//points to each of the LRU queue headers static item *tails[largest_id];//points to each LRU queue tail static unsigned int sizes[largest_id];// The number of item per LRU queue

You can see that the size of these three arrays is the same as the maximum slab class number provided by Slabclass. This also confirms that an LRU queue corresponds to a class of item. Heads[i] The first item,tails[i] pointing to Class I of the LRU list, which points to the last item,sizes[i of the Class I LRU list], indicates how many item is in the Class I LRU list. The structure consisting of the LRU queue and heads and tails is as shown:

Basic operation of the LRU queue:

Insert operation:

Assuming that we have an item to insert into the LRU list, you can insert the item into the LRU queue by calling the Item_link_q function. The following is the specific implementation code.

Insert item into the LRU queue's head static void Item_link_q (item *it) {/* Item is the new head */    item **head, **tail;    ASSERT (It->slabs_clsid < largest_id);    Assert ((It->it_flags & item_slabbed) = = 0);    Head = &heads[it->slabs_clsid];    tail = &tails[it->slabs_clsid];    ASSERT (it! = *head);    Assert ((*head && *tail) | | (*head = = 0 && *tail = = 0)); /head Interpolation Insert the item    it->prev = 0;    It->next = *head;    if (it->next) It->next->prev = it;    *head = it;//The item is the first node of the corresponding linked list//If the item is the first item on the corresponding ID, then it is also considered the last item//on that ID chain because the head interpolation method is used at head, so the first inserted item, In the back, it's really the last item    if (*tail = = 0) *tail = it;    sizes[it->slabs_clsid]++;//number plus a    return;}

Delete operation:

With the Insert function, there must be a corresponding delete function. Deleting a function is quite simple, mainly to deal with the deletion of this node after the node's front and rear drive node how to stitch together.

Remove it from the corresponding LRU queue by removing the static void Item_unlink_q (item *it) {    item **head, **tail;    ASSERT (It->slabs_clsid < largest_id);    Head = &heads[it->slabs_clsid];    tail = &tails[it->slabs_clsid];    if (*head = = it) {//The first node on a list        asserts (It->prev = = 0);        *head = it->next;    }    if (*tail = = it) {//The last node on the list        asserts (It->next = = 0);        *tail = it->prev;    }    ASSERT (It->next! = it);    ASSERT (It->prev! = it);//Connect the predecessor node of item with the back drive node    if (it->next) It->next->prev = it->prev;    if (it->prev) It->prev->next = it->next;    sizes[it->slabs_clsid]--;//number minus a    return;}

You can see whether an item is inserted or deleted, and its time-consuming is constant. In fact, for memcached, almost all of the operating time complexity is constant-level.

Update operation:

         Why do you want to insert the item into the LRU queue header? Of course, simplicity is one of the reasons for this. But more importantly, it's an LRU queue!! Remember the LRU inside the operating system. This is an elimination mechanism. In the LRU queue, the more you rank the less you use the item, the greater the chance of being eliminated. So fresh item (access time new), to be in the front of not so fresh item, so insert the LRU queue Head is the choice. This is supported by the Do_item_update function below. The Do_item_update function removes the old item from the LRU queue before inserting it into the LRU queue (at which point it is ranked first in the LRU queue). In addition to updating the item's position in the queue, it also updates the time member of item, which indicates the last visit (absolute time). If it is not for LRU, then the simplest implementation of the Do_item_update function is to update the time member directly.

#define ITEM_UPDATE_INTERVAL 60//Update frequency is 60 sec void Do_item_update (ITEM *it) {///The following code can see that the update operation is time-consuming. If this item is frequently accessed,//Then it causes too many update, too many time-consuming operations. At this point the update interval is created//. If the last access time (or the update time) is now (current_time)//is still within the update interval, it is not updated. The update was exceeded.    if (It->time < Current_time-item_update_interval) {        mutex_lock (&cache_lock);        if ((It->it_flags & item_linked)! = 0) {            item_unlink_q (it);//Remove            It->time = current_time;//from the LRU queue Update access Time            item_link_q (it);//INSERT into the LRU queue header        }        mutex_unlock (&cache_lock);}    }

When memcached processes the Get command, the Do_item_update function is called to update the access time of item, updating its location in the LRU queue. The Get command is a frequent command in memcached, and the first or last item in the LRU queue is more frequently get. For the first few of the item, calling Do_item_update is not very meaningful, because the call do_item_update after its position or the first few, and LRU obsolete more item also difficult to eliminate them (an LRU queue of the item number is many). On the other hand, the Do_item_update function is time-consuming and time-consuming because it is going to preempt cache_lock locks. If you call the Do_item_update function frequently, performance will degrade a lot. So memcached is using the update interval.

Here's how to insert and delete an item in the LRU queue, now tell me how to request an item from the slab allocator and how to return the item to the slab allocator.

Item Structure Body:

Although the item has been mentioned many times before, it is still a blur for the item to grow. Let's take a look at the full definition of the item structure, and pay attention to the English comments.

#define ITEM_LINKED 1//The item is inserted into the LRU queue by # define ITEM_CAS 2///The item uses Cas#define item_slabbed 4//The item is also in Slab's free queue and is not assigned    #define ITEM_FETCHED 8//The ITEM is inserted into the LRU queue and is accessed by the worker thread over the typedef struct _STRITEM {struct _stritem *next;//next pointer for the LRU linked list struct _stritem *prev;//prev pointer for the LRU list struct _stritem *h_next;//h_next pointer for the hash table conflict chain rel_time_t time;//last access time 。 Absolute time rel_time_t exptime;//expiration time, absolute time int nbytes;//The length of the data stored in this item unsigned short refcount;// The reference number of item uint8_t nsuffix;//suffix length */* Length of flags-and-length String */uint8_t It_flags;//item Genus       Sex/* item_* above */uint8_t slabs_clsid;/* which slab class we ' re in */uint8_t nkey;//key value length /* Key length, w/terminating null and padding */* This odd type prevents type-punning issues if we do * the L Ittle Shuffle to save space is not using CAS.        */union {uint64_t CAs;    Char end;    } data[]; /* If It_flags & Item_cas we Have 8 bytes CAS */* then null-terminated key */* then "flags length\r\n" (no terminating null) */* Then Data with terminating \ r \ n (no terminating null; it ' s binary!) */} item;

The English comment in the above code illustrates the item layout. The last member of the item struct is data[], which is defined as a flexible array. One use of a flexible array is that the data field is stored behind the array. Memcached's item also uses a flexible array. The above item simply defines the member of the item structure itself, but the previous post has always used item to represent the data stored by item. It is reasonable to write like this. Because the item structure itself and the item corresponding to the data are stored in the slab allocator allocation of the same piece of memory inside. When the slab memory allocation initializes the Slabclass array, its allocated memory block size is sizeof (item) +settings.chunk_size.

The item struct is followed by a bunch of data, not just data that the user wants to store, as shown in the following:

Looking at the diagram above may not be entirely clear about the item and the storage of the corresponding data. I understand that code-farmers need codes to be completely clear.

#define ITEM_KEY (item) ((Char* & (item)->data) + ((item)->it_flags & Item_cas)? sizeof (uint64_t ): 0)) #define ITEM_SUFFIX (item) ((char*) & (item)->data) + (item)->nkey + 1 + ((item)->it_flags &A mp Item_cas)? sizeof (uint64_t): 0)) #define ITEM_DATA (item) (Char* & (item)->data) + (item)->nkey + 1 + (item) Nsuffix + ((item)->it_flags & Item_cas)? sizeof (uint64_t): 0)) #define ITEM_NTOTAL (item) (sizeof (struct _stritem) + (item)->nkey + 1 + (item)->nsuffix + (item)->nbytes + ((item)->it_flags & I Tem_cas)?                     sizeof (uint64_t): 0)) static size_t Item_make_header (const uint8_t nkey, const int flags, const int Nbytes, Char *suffix, uint8_t *nsuffix) {/* suffix are defined at + chars elsewhere: */*nsuffix = (uint8_t) snprint    F (suffix, +, "%d%d\r\n", flags, nbytes-2); return sizeof (item) + Nkey + *nsuffix + nbytes;//calculate total size}//key, flags, EXptime three parameters are the parameters that users enter when they use the set, add command to store a single piece of data. Nkey is the length of the key string. Nbytes is the data length +2 that the user wants to store because the "\ r \ n"//CUR_HV at the end of data is the hash value computed from the key value key. Item *do_item_alloc (char *key, const size_t nkey, const int flags, const rel_time_t EXPTIME, const int    nbytes, const uint32_t CUR_HV) {uint8_t nsuffix;    Item *it = NULL; Char suffix[40];//The total space required to store this item. Note that the first parameter is nkey+1, so the above macros are calculated when//Used (item)->nkey + 1 size_t ntotal = Item_make_header (nkey + 1, flags, nbytes, suffix, &    Amp;nsuffix);    if (Settings.use_cas) {//Opens the CAS function ntotal + = sizeof (uint64_t);    }//According to size judging from which slab unsigned int id = slabs_clsid (ntotal); if (id = = 0)//0 means that it does not belong to any one slab return 0;...it = Slabs_alloc (ntotal, id);//Request memory slab from it->refcount allocator = 1;it->it_ Flags = Settings.use_cas?    item_cas:0;    It->nkey = Nkey;    It->nbytes = nbytes;    memcpy (Item_key (it), key, nkey);//Only Nkey bytes are copied here, and the last byte is empty it->exptime = Exptime; memcpy (Item_suffix (it), suffix, (size_t) NsuffIX); It->nsuffix = Nsuffix;return it;}

Code farmers have a good understanding of the above code of the several macro definitions and Item_make_header functions. The above code is a super-simple introduction to the Do_item_alloc function, the reason is super simple because this function is actually quite complex.

The above simply gives the slab to apply for an item, and now post the code to return the item.

items.c file void Item_free (item *it) {    size_t ntotal = item_ntotal (it);    unsigned int clsid;    CLSID = it->slabs_clsid;    It->slabs_clsid = 0;    Slabs_free (it, ntotal, CLSID);} slabs.c file void Slabs_free (void *ptr, size_t size, unsigned int id) {    pthread_mutex_lock (&slabs_lock);    Do_slabs_free (PTR, size, id);//Return to Slab allocator    pthread_mutex_unlock (&slabs_lock);}

Item and hash table contact:

The previous Do_item_alloc function is to request an item based on the desired size. From the Do_item_alloc implementation code, it does not insert this item into the hash table and the LRU queue. In fact, this task is implemented by another function.

Next look at how to pass the item to the hash table, the LRU queue, and how to retract the item from the hash table, the LRU queue, and sometimes return it to slab.

Insert item into hash table and LRU queue, insert into hash table requires hash value Hvint Do_item_link (item *it, const uint32_t HV) {//Make sure this item has been allocated from slab and not inserted into the LRU queue Assert (It->it_flags & (item_linked|    item_slabbed) = = 0); When the hash table is not migrating data for the extension, it is inserted into the hash table item//when the hash table migrates the data, it occupies the lock.    Mutex_lock (&cache_lock);    It->it_flags |= item_linked;//joined the link flag it->time = current_time; /* Allocate a new CAS ID on link.    */Item_set_cas (it, (Settings.use_cas)? get_cas_id (): 0); Assoc_insert (it, HV);//insert this item into the hash table item_link_q (it);//insert this item into the list REFCOUNT_INCR (&it->refcount);//    Reference count plus one mutex_unlock (&cache_lock); return 1;}    Delete from hash table, so need hash value hvvoid Do_item_unlink (item *it, const uint32_t HV) {mutex_lock (&cache_lock); if ((It->it_flags & item_linked)! = 0) {it->it_flags &= ~item_linked;//minus the link flag assoc_delete (Item_key (it), It->nkey, HV);//Remove this item from the hash table item_unlink_q (it);//Remove the item do_item_remove (it) from the list;//To Slab Also this item} mutex_unlock (&cachE_lock);}    void Do_item_remove (item *it) {Assert ((It->it_flags & item_slabbed) = = 0);    ASSERT (It->refcount > 0); if (REFCOUNT_DECR (&it->refcount) = = 0) {//reference count equals 0 when return item_free (it);//Return the item to slab}}

When you use the Do_item_remove function to return an item to slab, the reference number of the item is tested to be equal to 0. The number of references can be simply understood as there is no worker thread in the use of this item. The number of references will be explained in detail later in the blog post.

The lock is rarely seen in the preceding code. In practice, however, the previous item operation needs to be locked because multiple worker threads can manipulate both the hash table and the LRU queue. Locks are rarely seen because they all use parcel functions (this concept should be familiar if you read UNIX network programming). Lock and unlock in the wrapping function. In the preceding functions, the function names are generally prefixed with do_. Its corresponding package function name is to remove the prefix do_. The introduction of the lock is not a Ben Boven task, there will be a special post to introduce the use of locks.

memcached Source Analysis-----LRU Queue and item structure