Memcached is an efficient distributed memory cache. It understands the memory management mechanism of memcached, so that we can better understand memcached and optimize our data features so that it can be used by me. Here, I will briefly talk about some of my knowledge about memcached memory management. Without a special note, the memcached mentioned here is version 1.2, and there are some differences between version 1.1 and version 1.2.
Basic concepts: slab and chunkMemcached memory structure has two very important concepts: slab and Chunk. We first have a perceptual knowledge of these two concepts: Figure 1 memcached Memory Structure
Slab is a memory block, which is the smallest unit of memory applied for by memcached at a time. When starting memcached, the parameter-M is usually used to specify its available memory, but not all the memory is allocated at the moment of startup, an slab is applied only when necessary. The size of slab is fixed to 1 m (1048576 bytes). An slab is composed of several chunks of the same size. Each chunk stores an item struct, a pair of keys, and values. Although the chunk size in the same slab is equal, the chunk size in different slab is not necessarily equal. In memcached, the chunk size varies according to the chunk size, slab can be divided into many types (class ). When starting memcached, you can use-VV to view the slab type: $ memcached-VV
Slab Class 1: Chunk Size 80 perslab 13107
Slab Class 2: Chunk Size 104 perslab 10082
Slab Class 3: Chunk Size 136 perslab 7710
Slab Class 4: Chunk Size 176 perslab 5957
Slab Class 5: Chunk Size 224 perslab 4681
Slab Class 6: chunk size 280 perslab 3744
Slab Class 7: Chunk Size 352 perslab 2978
Slab Class 8: Chunk Size 440 perslab 2383
Slab Class 9: Chunk Size 552 perslab 1899
Slab class 10: Chunk Size 696 perslab 1506
Slab class 11: Chunk Size 872 perslab 1202
Slab Class 12: Chunk Size 1096 perslab 956
Slab class 13: Chunk Size 1376 perslab 762
Slab class 14: Chunk Size 1720 perslab 609
Slab class 15: Chunk Size 2152 perslab 487
Slab Class 16: Chunk Size 2696 perslab 388
Slab Class 17: Chunk Size 3376 perslab 310
Slab class 18: Chunk Size 4224 perslab 248
Slab class 19: Chunk Size 5280 perslab 198
Slab class 20: Chunk Size 6600 perslab 158
Slab Class 21: Chunk Size 8256 perslab 127
Slab Class 22: Chunk Size 10320 perslab 101
Slab Class 23: Chunk Size 12904 perslab 81
Slab class 24: Chunk Size 16136 perslab 64
Slab Class 25: Chunk Size 20176 perslab 51
Slab Class 26: Chunk Size 25224 perslab 41
Slab class 27: Chunk Size 31536 perslab 33
Slab Class 28: Chunk Size 39424 perslab 26
Slab class 29: Chunk Size 49280 perslab 21
Slab class 30: Chunk Size 61600 perslab 17
Slab class 31: Chunk Size 77000 perslab 13
Slab class 32: Chunk Size 96256 perslab 10
Slab Class 33: Chunk Size 120320 perslab 8
Slab Class 34: Chunk Size 150400 perslab 6
Slab class 35: Chunk Size 188000 perslab 5
Slab class 36: Chunk Size 235000 perslab 4
Slab Class 37: Chunk Size 293752 perslab 3
Slab Class 38: Chunk Size 367192 perslab 2
Slab class 39: Chunk Size 458992 perslab 2
Slab class 40: Chunk Size 573744 perslab 1
Slab class 41: Chunk Size 717184 perslab 1
Slab class 42: Chunk Size 1048576 perslab 1
Figure 2 Slab group information: by default, memcached divides slab into 40 categories (class1 ~ Class40), in class 1, the chunk size is 80 bytes, because the size of an slab is fixed 1048576 bytes (1 m ), therefore, class1 can contain a maximum of 13107 chunks: 13107 × 80 + 16 = 1048576. In class1, the remaining 16 bytes are not the size of a chunk (80 bytes ), therefore, it will be wasted. The size of each type of chunk has a certain formula. Assume that I represents a classification. The formula for calculating Class I is as follows: chunk size (Class I): (default_size + item_size) * f ^ (I-1) + chunk_align_bytes
- Default_size: the default size is 48 bytes, that is, the default size of key + value of memcached is 48 bytes, which can be adjusted by the-n parameter;
- Item_size: the length of the item struct, which is fixed to 32 bytes. Default_size is 48 bytes, and item_size is 32. Therefore, the chunk size of class1 is 48 + 32 = 80 bytes;
- F is a factor that determines the chunk variation. The default value is 1.25. Adjusting F can affect the chunk step size. You can use-F to specify the chunk size at startup;
- Chunk_align_bytes is a modified value to ensure that the chunk size is an integer multiple of a value (the chunk size must be an integer multiple of 4 on a 32-bit machine ).
From the above analysis, we can see that the actual parameters we can adjust include-F and-N. in the actual operation of memcached, we also need to observe our data features and properly adjust F, N value to make full use of our memory to reduce waste.
Memory Allocation requestMemcached memory management adopts pre-allocation and group management. Group Management is the slab class we mentioned above. slab is divided into many types according to chunk size. The following describes the memcached memory pre-allocation process. When an item is added to memcached, memcached first selects the most appropriate slab Class Based on the item size. For example, the size of the item is 190 bytes, by default, the chunk size of Class 4 is 160 bytes. The chunk size of Class 5 is 200 bytes, which is greater than 190 bytes, therefore, this item will be placed in Class 5 (it is inevitable that there will be 10 bytes of waste). After calculating the chunk to be put, memcached checks whether the chunks of this class are idle. If not, it applies for 1 m (1 slab) space and divides it into chunks of this type. For example, when we put a 190-byte item into memcached for the first time, memcached will generate an slab Class 2 (also called a page) and use a chunk, the remaining 5241 chunks will be used for the next time there will be a suitable size item. When we use all the 5242 chunks, there will be another chunks in the range of 160 ~ When an item between 200 bytes is added, memcached will generate a Class 5 slab again (so there are two pages ). To view slab usage, Telnet the IP port and enter the stats slabs command, for example, Telnet 10.0.4.210 11211.
| Stats slabs Stat 5: chunk_size 200 Stat 5: chunks_per_page 5242 Stat 5: total_pages 1 Stat 5: total_chunks 5242 Stat 5: used_chunks 5242 Stat 5: free_chunks 0 Stat 5: free_chunks_end 5241 stat active_slabs 1 stat total_malloced 1048400 |
Figure 3 stats slab Figure 3 shows the statistical results after a 190-byte item is placed for the first time. Conclusion: I understand that the M parameter at the beginning of memcached startup is used to determine the size of memory allocated. It is allocated if it is not started .. Instead, it is dynamically allocated when memcached is used. When you set an item, memcached will judge the size of each slab added based on the item size. It is different when slab of different class classes are assigned chunk... You can optimize your memcached by Using echo stats sizes | NC 127.0.0.1 11211 | less to view the size distribution of most items in the memory .. You can adjust the slab of different classes .. Please refer to the following link for more information: http://blog.sina.com.cn/s/blog_6ab686dc01010169.htmlwhen memcached allocates memory.
