memcached comprehensive analysis of –2. Understanding memcached Memory Storage _php Tutorial

The following is the second part of the comprehensive anatomy of memcached.

Release Date: 2008/7/9
Original link: http://gihyo.jp/dev/feature/01/memcached/0002

Links to this series of articles are here:

• 1th Time: http://www.phpchina.com/html/29/n-35329.html
• 2nd time: http://www.phpchina.com/html/30/n-35330.html
• 3rd time: http://www.phpchina.com/html/31/n-35331.html
• 4th Time: http://www.phpchina.com/html/32/n-35332.html
• 5th time: http://www.phpchina.com/html/32/n-35333.html

• Slab allocation mechanism: Defragment memory for reuse
  • Main terms of Slab allocation
• The principle of caching records in slab
• Disadvantages of Slab Allocator
• Tuning with growth factor
• View the internal state of the memcached
• View the usage status of slabs
• Summary of memory storage

I am the former mixi of the research and Development Group of the corporation. The last article described the memcached as a distributed cache server. This article will introduce the implementation of the internal structure of memcached and how the memory is managed. In addition, weaknesses caused by the internal structure of the memcached will also be explained.

Slab allocation mechanism: Defragment memory for reuse

The most recent memcached by default uses a mechanism called slab allocator to allocate and manage memory. Prior to the advent of this mechanism, the allocation of memory was performed simply by malloc and free for all records. However, this approach can lead to memory fragmentation, aggravating the burden on the operating system memory manager, and in the worst case, cause the operating system to be slower than the memcached process itself. Slab Allocator was born to solve the problem.

Let's look at the principle of slab allocator. The following is the goal of slab allocator in the memcached documentation:

The primary goal of the slabs subsystem in memcached is to eliminate memory fragmentation issues totally by using fixed-s Ize memory chunks coming from a few predetermined size classes.

In other words, the basic principle of the Slab allocator is to divide the allocated memory into blocks of a specific length in a predetermined size to completely resolve the memory fragmentation problem.

The principle of Slab allocation is quite simple. Divide the allocated memory into blocks of various sizes (chunk) and divide the same size blocks into groups (the Chunk collection) (Figure 1).

Fig. 1 structure diagram of Slab allocation

Also, slab allocator has the purpose of reusing allocated memory. In other words, the allocated memory is not freed, but reused.

Main terms of Slab allocation

Page

The memory space allocated to slab, which is 1MB by default. After assigning to slab, the slab is divided into chunk according to the size of the.

Chunk

The memory space used to cache records.

Slab Class

A group of chunk of a specific size.

The principle of caching records in slab

The following shows how memcached selects slab and caches the data sent by the client to chunk.

Memcached based on the size of the data received, select the slab that best fits the data size (Figure 2). Memcached holds a list of idle chunk within slab, selects chunk based on the list, and caches the data in it.

Figure 2 How to select a group to store records

In fact, Slab allocator also has advantages and disadvantages. Here's a look at its drawbacks.

Disadvantages of Slab Allocator

Slab allocator solved the original memory fragmentation problem, but the new mechanism also brought new problems to memcached.

The problem is that the allocated memory cannot be effectively exploited because it allocates memory of a specific length. For example, by caching 100 bytes of data into a 128-byte chunk, the remaining 28 bytes are wasted (Figure 3).

Figure 3 Use of chunk space

There is no perfect solution for this problem, but the more effective solution is documented in the documentation.

The most efficient-on-the-waste-to-use-a list of size classes that closely matches (if that's at all Possibl e) Common sizes of objects, the clients of this particular installation of memcached is likely to store.

That is, if you know in advance the common size of the data sent by the client, or if you only cache data of the same size, you can reduce waste if you use a list of groups that fit the data size.

Unfortunately, it is not possible to perform any tuning at this time, only to look forward to future versions. However, we can adjust the difference in the size of the slab class. Next, the growth factor option is described.

Tuning with growth factor

memcached specifies the growth factor factor at startup (with the-f option) to control the difference between slab to some extent. The default value is 1.25. However, before this option occurs, this factor was once fixed to 2, called the "Powers of 2" policy.

Let's try using the previous settings to start memcached in verbose mode:

$ memcached-f 2-VV

The following is the verbose output after startup:

Slab class 1:chunk size Perslab 8192 slab class 2:chunk size Perslab 4096 slab class 3:chunk size Perslab 2 048 Slab class 4:chunk size 1024x768 perslab 1024x768 Slab class 5:chunk size 2048 perslab Slab class 6:chunk size 4096 per Slab Slab class 7:chunk size 8192 perslab Slab class 8:chunk size 16384 Perslab * Slab class 9:chunk size 3276 8 Perslab Slab class 10:chunk size 65536 perslab Slab class 11:chunk size 131072 Perslab 8 slab class 12:chunk Si Ze 262144 perslab 4 slab class 13:chunk size 524288 Perslab 2

As you can see, starting with a 128-byte group, the size of the group increases to twice times the original. The problem with this setting is that the difference between slab is large, and in some cases it is quite a waste of memory. Therefore, to minimize memory waste, two years ago this option was appended with growth factor.

Take a look at the current default settings (f=1.25) when the output (space limit, this is only written to the 10th group):

Slab class 1:chunk size Perslab 11915 Slab class 2:chunk size 7 Perslab 9362 Slab class 3:chunk size 144 Perslab 281 Slab class 4:chunk size 184 perslab 5698 Slab class 5:chunk size 232 Perslab 4519 Slab class 6:chunk size 296 persl AB 3542 Slab class 7:chunk size 376 Perslab 2788 Slab class 8:chunk size 472 Perslab 2221 Slab class 9:chunk size 592 p Erslab 1771 Slab class 10:chunk size 744 Perslab 1409

As can be seen, the gap between groups is much smaller than the factor of 2 o'clock, which is more suitable for caching hundreds of-byte records. From the above output, you may find some calculation errors, which are deliberately set to keep the number of bytes aligned.

When introducing memcached into a product or deploying it directly using default values, it is best to recalculate the expected average length of the data and adjust the growth factor to get the most appropriate settings. Memory is a precious resource, and a waste of it is too bad.

Here's how to use memcached's stats command to see a wide variety of information such as slabs utilization.

View the internal state of the memcached

Memcached has a command called stats that can be used to obtain a wide variety of information. There are many ways to execute commands, with Telnet the simplest:

$ telnet Host name Port number

After connecting to memcached, enter stats and press ENTER to get a variety of information including resource utilization. Also, enter "stats slabs" or "stats items" to get information about the cache record. To end the program, enter quit.

The details of these commands can refer to the Protocol.txt documentation within the memcached package.

$ telnet localhost 11211 Trying:: 1 ... Connected to localhost. Escape character is ' ^] '. Stats stat PID 481 stat uptime 16574 stat time 1213687612 stat version 1.2.5 stat pointer_size stat Rusage_user 0.10229 7 stat rusage_system 0.214317 stat curr_items 0 stat total_items 0 Stat bytes 0 Stat curr_connections 6 stat total_connect Ions 8 stat connection_structures 7 stat cmd_get 0 Stat cmd_set 0 stat get_hits 0 Stat get_misses 0 Stat Evictions 0 Stat Bytes_read Stat bytes_written 465 stat limit_maxbytes 67108864 STAT threads 4 END quit

Additionally, if you install libmemcached, the client library for the C + + language, the MemStat command will be installed. Using the method is simple, you can get the same information as Telnet with fewer steps, and you can get information from multiple servers at once.

$ memstat--servers=server1,server2,server3,...

Libmemcached can be obtained from the following address:

• Http://tangent.org/552/libmemcached.html

View the usage status of slabs

Using memcached, a Perl script named Memcached-tool, written by Brad, makes it easy to get slab usage (IT organizes memcached return values into easy-to-read formats). You can get the script from the following address:

• Http://code.sixapart.com/svn/memcached/trunk/server/scripts/memcached-tool

The method of use is also extremely simple:

$ memcached-tool Host Name: Port option

You do not need to specify options when viewing slabs usage, so you can use the following command:

$ memcached-tool Host Name: Port

The information obtained is as follows:

# item_size Max_age 1mb_pages Count full? 1 104 B 1394292 s 1215 12249628 Yes 2 136 B 1456795 s 400919 Yes 3 176 b 1339587 s 196567 Yes 4 224 b 1360926 s 109  510221 Yes 5 280 b 1570071 s 183452 Yes 6 352 b 1592051 s 229197 Yes 7 [b 1517732 S] 157183 Yes 8 552 b 1460821 S 117697 Yes 9 696 b 1521917 s 143 215308 Yes 872 B 1695035 s 205 246162 Yes 1.1 KB 1681650 s 233 221968 Yes 12 1.3 KB 1603363 s 241 183621 Yes 1.7 kb 1634218 S 94 57197 Yes 2.1 KB 1695038 s 36488 Yes 2.6 kb 1747075 S 65 2 5203 Yes 3.3 KB 1760661 s 24167 Yes

The meanings of each column are:

Column meaning #slab class number Item_sizechunk size Max_agelru The oldest record in the lifetime 1mb_pages the number of records in slab that are assigned to Countslab full? Whether the slab contains idle chunk

The information obtained from this script is very convenient for tuning and is highly recommended.

Summary of memory storage

This paper simply explains the caching mechanism and tuning method of memcached. I hope readers can understand the principle of memcached memory management and its advantages and disadvantages.

Next time, we will continue to explain the principles of LRU and expire, as well as the latest development direction of memcached-expandable system (pluggable Architecher).

copyright Notice : Can be reproduced arbitrarily, but must be reproduced when the original author Charlee, the original link and this statement.

http://www.bkjia.com/PHPjc/735133.html www.bkjia.com true http://www.bkjia.com/PHPjc/735133.html techarticle The following is the second part of the comprehensive anatomy of memcached. Posted: 2008/7/9 original link: http://gihyo.jp/dev/feature/01/memcached/0002 the link to this series of articles is here: ...

