memcached Introduction and related knowledge

memcached Introduction

1 , Memcached is a free, open-source, high-performance cache system with distributed memory objects. memcached maintains a unified, huge hash table in memory that can be used to store data in a variety of formats, including images, videos, files, and the results of database retrieval.

2 , it improves the speed of dynamic, database-driven Web sites by caching data and objects in memory to reduce the number of times the database is read. In-Memory cached data is accessed through the API.
3,Memcached based on a key storage key /value value pairs of hAshmap .
4, the Guardian process (daemon ) is written in C ,
5, however, the client can be written in any language and communicate with the daemon through the memcached protocol.

1, the characteristics of memcached

(1), the protocol is simple: Memcached is based on the broad line of the protocol, can be directly through Telnet to the memcached server access to data.

(2), Libevent-based event processing: Libevent is a library of C development, it will be BSD system Kqueue,linux system Epoll and other event processing functions encapsulated into an interface, to ensure that even the server side of the increase in the number of connections can also play a good performance. Memcached uses this library for asynchronous event processing.

(3), built-in memory management method: Memcached a set of their own management of memory, the management is very efficient, all the data are stored in the memcached built-in memory, when the data deposited in full space, using the LRU algorithm automatically delete unused cache, That is, the memory space for reusing outdated data. Memcached is designed for the cache system, without taking into account the data disaster-tolerant problem, and the machine's memory, restart the machine data will be lost.

(4), non-communication between the memcached has distribution characteristics: Each memcached server between each other, are independent access to data, do not share any information. Through the design of the client, let memcached have distributed, can support massive cache and large-scale application.

2, the application of memcached business scenarios?

1) If the site contains a dynamic Web page with a large amount of traffic, the load on the database will be high. Because most database requests are read, memcached can significantly reduce the database load.

2) If the load on the database server is low but the CPU usage is high, you can cache the computed results (computed objects) and the rendered page template (enderred templates).

3) Use memcached to cache session data and temporary data to reduce write operations to their databases.

4) cache some small but frequently accessed files.

5) caching the results of web ' services ' (non-IBM advertised Web services, translator notes) or RSS feeds:

3, not applicable to memcached business scenarios?

1) The size of the cache object is greater than 1MB

Memcached itself is not designed to handle huge multimedia (large media) and huge binary blocks (streaming huge blobs).

2) key is longer than 250 characters

3) Virtual host does not let run memcached service

If the application itself is hosted on a low-end virtual private server, such virtualization technologies like VMware are not suitable for running memcached. Memcached needs to take over and control large chunks of memory, and if memcached managed memory is swapped out by the OS or hypervisor, the performance of the memcached will be compromised.

4) application running in an unsafe environment

Memcached to provide any security policy, the memcached can be accessed only via Telnet. If your app is running on a shared system, you need to focus on security issues.

5) The business itself needs to be persistent data or need to be database

4. Unable to traverse all the item in memcached

This operation is relatively slow and blocks other operations (here, slower than memcached other commands). Memcached all non-debug (non-debug) commands, such as add, set, Get, Fulsh, and so on, regardless of how much data is stored in memcached, their execution consumes only constant time. The amount of time spent executing a command that traverses all of the item will increase as the volume of data in the memcached increases. Blocking occurs when other commands cannot be executed because of waiting (the command to traverse all item finishes).

5. The maximum length of a key that memcached can accept is 250 characters

The maximum length of a key that memcached can accept is 250 characters. It is important to note that 250 is a limitation within the memcached server side. If you are using a memcached client that supports "key prefixes" or similar features, the maximum length of a key (prefix + original key) can be more than 250 characters. It is recommended to use shorter keys, which saves memory and bandwidth.

6. The size of a single item is limited to 1M bytes

Because this is the algorithm for the memory allocator.

A detailed answer:

1) memcached memory storage engine, use slabs to manage memory. Memory is divided into slabs chunks of unequal size (first divided into slabs of equal size, then each slab is divided into equal size chunks, slab of different chunk size is unequal). The size of the chunk starts with a minimum number, and grows by a factor until the maximum possible value is reached. If the minimum value is 400B, the maximum value is 1MB, the factor is 1.20, and the size of each slab chunk is:

slab1-400b;slab2-480b;slab3-576b. Slab the larger the chunk, the greater the gap between it and the slab ahead. Therefore, the larger the maximum value, the less memory utilization. Memcached must pre-allocate memory for each slab, so if you set a smaller factor and a larger maximum value, you will need to provide more memory for memcached.

2) do not attempt to access large data in memcached, such as putting huge web pages into mencached. Because it takes a long time to load and unpack big data into memory, the performance of the system can be poor. If you do need to store more than 1MB of data, you can modify the value of Slabs.c:power_block, and then recompile memcached, or use an inefficient malloc/free. In addition, the database, mogilefs and other alternatives can be used instead of the memcached system.

7. How does the memcached memory allocator work? Why not apply malloc/free!? Why use slabs?

In fact, this is a compile-time option. The internal slab allocator is used by default, and the built-in slab allocator should indeed be used. At the earliest, memcached only used Malloc/free to manage memory. However, this approach does not work well with the memory management of the OS before. Repeated malloc/free caused memory fragmentation, and the OS eventually spent a lot of time looking for contiguous blocks of memory to meet malloc requests, rather than running the memcached process. The slab dispenser was born to solve the problem. The memory is allocated and divided into chunks, which has been reused. Because memory is divided into slabs of different sizes, if the size of the item is not appropriate for the slab that is chosen to store it, some memory is wasted.

8. What are the restrictions on the expiration time of item memcached?

The item object can expire up to 30 days in length. Memcached the incoming Expiration time (time period) is interpreted as a point in time, once at this point in time, memcached the item to a failed state, which is a simple but obscure mechanism.

9, what is the binary protocol, do you need to pay attention?

The binary protocol attempts to provide a more efficient and reliable protocol for the end, reducing the CPU time generated by the client/server side due to processing protocols. According to Facebook's tests, parsing the ASCII protocol is the most CPU-intensive part of memcached.

10. How does the memcached memory allocator work? Why not apply malloc/free!? Why use slabs?

In fact, this is a compile-time option. The internal slab allocator is used by default, and the built-in slab allocator should indeed be used. At the earliest, memcached only used Malloc/free to manage memory. However, this approach does not work well with the memory management of the OS before. Repeated malloc/free caused memory fragmentation, and the OS eventually spent a lot of time looking for contiguous blocks of memory to meet malloc requests, rather than running the memcached process. The slab dispenser was born to solve the problem. The memory is allocated and divided into chunks, which has been reused. Because memory is divided into slabs of different sizes, if the size of the item is not appropriate for the slab that is chosen to store it, some memory is wasted.

11. Is the memcached atomic?

All the individual commands that are sent to the memcached are completely atomic. If you send a set command and a GET command for the same data at the same time, they do not affect each other. They will be serialized and executed successively. Even in multithreaded mode, all commands are atomic. However, the command sequence is not atomic. If you first get an item with a GET command, modify it, and then set it back to memcached, the system does not guarantee that the item is not manipulated by another process (process, not necessarily an operating system). Memcached 1.2.5 and later, the Get and CAS commands are available, and they solve the problem above. If you use the GET command to query a key, Item,memcached returns a unique identifier for the item's current value. If the client program overwrite this item and want to write it back to memcached, you can send that unique identity together with the memcached by using the CAS command. If the item's unique identity in the memcached is consistent with what you provide, the write operation will succeed. If the item is also modified by another process during this time, the unique identity of the item stored in the memcached will change and the write operation will fail.

12, memcached data compression problem? What is the compression ratio?

The memcached single item stores up to 1m of data. The cache key can be encrypted with MD5 for two reasons, 1 because the memcached has a limit on the length of the key, and if the limit length is exceeded, the item will remain in the failed state for 2 because of a security issue.

With PHP Pecl Memcache's buddies should know, Memcache client's add and set methods in the third parameter is to indicate whether to use compressed storage

BOOL Memcache::add (string key, mixed var [, int flag [, int expire]]) bool Memcache::set (string key, mixed var [, int flag [, int expire]])

Flag memcache_compressed to store the item compressed (uses zlib).

There is also a more important function, is to set the threshold to be compressed

Threshold
min_saving Specifies the minimum amount of savings to actually store the value compressed. The supplied value must be between 0 and 1. Default value is 0.2 giving a minimum 20% compression savings.

Wherein, threshold is set to start the minimum length of compression, larger than the length of the string will try to automatically compress;
Min_saving is the minimum percentage of savings using compressed storage. Compressed storage is used when the length < original length * (1-min_saving) is compressed.

13, memcached If there is a machine down, how to do?

Memcache itself does not realize the cluster function, if you want to use the MEMCAHCE to realize the cluster need third-party software or design their own implementation of programming, here will use Memagent Proxy implementation, Memagent also known as magent, attention, do not use both as two tools. (Original address: http://blog.csdn.net/liu251890347/article/details/38414247)

It can be seen that there are two magent nodes, two memcached nodes, each magent node is represented by two memcached nodes respectively, and the application system side uses Magent pool to call memcache for storage. The hardware structure is two Linux servers, each server installed magent and memcached services, and set to boot. The advantage of this is that any one server goes down without affecting the magent pool to get memcache information, which is memcached high availability (HA).

1, the value of the connection pool through the magent will exist magent agent of all memcached up

2, if there is a memcached down through the Magent proxy way can also fetch value

3, if the memcached repair after the restart through the Magent proxy method to the value will be null, this is because the value of the memcache after the restart after the stop of the Memcache service disappears (because in memory), However, the magent is assigned to a machine by hashing the key, and the memcache will be taken from the machine after the reboot, and all the values taken will not be available.

Solutions

1, after each memcache downtime repair can write a program to all the other memcache in the cluster to copy all the information to the current downtime repaired memcache.

2, write their own agent, when the value from a memcached service is null, and then go to the other memcached to take the value.

Solution 2:memcached+repcached replication solves a single point of failure. (Original address: http://www.rootop.org/pages/2475.html)

Use repcached to do memcached master-slave redundancy. Repcached realizes the function of memcached replication, it is a single master single slave scheme, but master/slave are both readable and writable, and can be synchronized with each other, if the master bad slave detect the connection is broken, It will automatically listen and become master, and if Slave is broken, master will detect the connection and it will re-listen wait for the new slave to join.

memcached Introduction and related knowledge