Memcache memory usage

Since I took over a new project last week, this project is very dependent on memcache, so I have to begin to learn more about the memory usage of memcache. Here I will summarize my personal gains, it is also a small memcache optimization.

I. memcache Memory Allocation Mechanism

There are many explanations on this mechanism. My personal summary is as follows.

1. Page is the minimum unit of memory allocation.

   Memcached memory is allocated in page units. By default, the next page is 1 MB. You can specify it at startup using the-I parameter. If you need to apply for memory, memcached will divide a new page and allocate it to the desired slab area. Once the page is assigned, it will not be recycled or reassigned before restart (page ressign has been removed from version 1.2.8)
   

2. Slabs divides data space.

   Instead of putting all the data in size together, memcached divides the data space into a series of slabs in advance, and each slab is only responsible for data storage within a certain range. For example, each slab only stores data larger than or equal to the size of the previous slab. For example, slab 3 only stores data of 137 to 224 bytes. If the data size is 230byte, it will be allocated to slab 4. It can be seen that each slab is responsible for a different amount of space. By default, the maximum value of the next slab in memcached is 1.25 times that of the previous slab, you can modify the-F parameter to modify the growth ratio.
   

3. Chunk is the unit for storing cached data.

   Chunk is a series of fixed memory space, which is the maximum storage size of the slab that manages it. For example, all chunks of slab 1 are 104 bytes, while all chunks of slab 4 are 280 bytes. Chunk is the place where memcached actually stores cached data. Because the chunk size is fixed to the maximum value that slab can store, all data allocated to the current slab can be saved by chunk. If the data size of the time is smaller than the chunk size, the free space will be idle. This is designed to prevent memory fragments. For example, if the chunk size is 224 bytes and the stored data is only 200 bytes, the remaining 24 bytes will be idle.
   

4. Slab Memory allocation.

   Memcached specifies the maximum memory usage through-m at startup, but this will not be used at startup, and will be allocated to each slab as needed.
   If a new cache data is to be stored, memcached first selects a suitable slab and then checks whether the slab has idle Chunk. If yes, it is stored directly; if not, apply. When applying for memory, slab uses page as the unit. Therefore, a page of 1 MB is allocated to the slab no matter the size of the first data. After applying for a page, slab splits the page's memory by the chunk size, which becomes a chunk array, and selects a chunk array to store data. For example, slab 1 and slab 2 both allocate a page and split it into chunk arrays based on their respective sizes.
   

5. Memcached memory allocation policy.

   Based on the above introduction, memcached's memory allocation policy is to allocate pages based on slab requirements, and each slab uses chunk storage as needed.
   Note the following features,

   1. The pages allocated by memcached will not be recycled or re-allocated.
   2. Memory applied for by memcached will not be released
   3. Slab idle chunk will not be lent to any other Slab

 

After knowing this, we can understand why memcached has lost cache data when the total memory is not fully occupied.

 

-P <num> listening TCP port (default: 11211)-u <num> UDP listening port (default: 11211, disabled when 0) -D run in daemon mode-u <username> RUN memcached account non-root user-M <num> maximum memory usage unit is MB default is 64 MB-C <num> The default number of soft connections is 1024-v, which outputs warning and error messages.-VV: prints client requests and returned information.-H: prints help information.-I: prints the copyright information of memcached and libevent.-L <ip_addr> binding address (default: all allow, regardless of the Intranet or the local machine to change the IP address, there is a security risk, if set to 127.0.0.1, only local access is allowed)-P <File> write PID to file <File>, in this way, the backend process can be terminated quickly. You need to use-I override the size of each slab page with-D. adjusts Max item size (default: 1 MB, Min: 1 K, Max: 128 M)-N <bytes> minimum space allocated for key + value + flags (default: 48)

Connection: Telnet 127.0.0.1 11211.

Stats \ r \ n
Here we can see the number of times memcache is obtained, the current number of connections, the number of writes, the hit rate, and so on;

PID: process ID
Uptime: total running time, in seconds
Time: Current Time
Version: Version Number
......
Curr_items: Number of keyvalues in the current Cache
Total_items: Total number of cached keyvalues
Bytes: The amount of memory used by all caches
Current number of curr_connections connections
....
Pai_get: Total number of times of obtaining
Performance_set: Total number of writes
Get_hits: Total number of hits
Miss_hits: number of failed requests
.....
Bytes_read: Total number of read traffic bytes
Bytes_written: Total write traffic bytes
Limit_maxbytes: maximum memory usage, in bytes

Stats Reset
Clear statistics

Stats malloc
Display memory allocation data

Stats cachedump slab_id limit_num
Displays the list of the first limit_num keys in an slab. The format is as follows:
Item key_name [value_length B; expire_time | access_time S]
Memcached 1.2.2 and earlier versions show the access time (timestamp)
1.2.4 or later versions, including 1.2.4 display expiration time (timestamp)
If it is a key that never expires, expire_time is displayed as the start time of the server.

 

Set <key> <flags> <exptime> <bytes> \ r \ n <data block> \ r \ n
Set aaa1 33 3600 100
Aaaaaaa

Stats cachedump 7 2
Itemcopy_test1 [250 B; 1207795754 S]
Item copy_test [248 B; 1207793649 S]

Stats Slabs
Displays information about each slab, including the chunk size, quantity, and usage.

Stat 1: chunk_size 96
Stat 1: chunks_per_page 10922
Stat 1: total_pages 1
Stat 1: total_chunks 10922
Stat 1: used_chunks 3
Stat 1: free_chunks 1
Stat 1: free_chunks_end 10918
Stat 1: mem_requested 256
Stat 1: get_hits 262
Stat 1: performance_set 426
Stat 1: delete_hits 0
Stat 1: incr_hits 0
Stat 1: decr_hits 0
Stat 1: cas_hits 0
Stat 1: cas_badval 0

Stats items
Display the number of items in each slab and the age of the oldest item (the number of seconds from the last visit)

Stats detail [ON | off | dump]
Set or display detailed operation records

The parameter is on.
The parameter is off. Disable detailed operation records.
The parameter is dump, and detailed operation records are displayed (the number of times each key value is get, set, hit, and del)

8. Clear all key values
Flush_all
Note: flush does not delete items, but marks all items as expired. Therefore, memcache still occupies all memory.

8. Exit
Quit \ r \ n

 

 

$ Memcached-tool host name: the port obtains the following information: # item_size max_age 1mb_pages count full? 1 104 B 1394292 s 1215 12249628 Yes 2 136 B 1456795 s 52 400919 Yes 3 176 B 1339587 s 33 196567 Yes 4 224 B 1360926 s 109 510221 Yes 5 280 B 1570071 s 49 183452 Yes 6 352 B 1592051 s 77 229197 Yes 7 440 B 1517732 s 66 157183 Yes 8 552 B 1460821 s 62 117697 Yes 9 696 B 1521917 s 143 215308 yes10 872 B 1695035 s 205 yes11 246162 KB 1681650 s 233 221968 yes12 1.3 kb 1603363 s 241 183621 yes13 1.7 KB 1634218 s 94 57197 yes14 2.1 kb 1695038 S 75 36488 yes15 2.6 kb 1747075 S 65 25203 yes16 3.3 kb 1760661 s 78 24167 yes the meaning of each column is: column description # Slab class no. item_size chunk size max_age LRU oldest record survival time 1mb_pages assigned to slab page count slab record count full? Whether the slab contains idle chunk