Redis Data Summary (ix) virtual memory

First, the following Redis virtual memory and OS virtual memory is not matter, but the idea and purpose are the same. is to temporarily swap infrequently accessed data from memory to disk, freeing up valuable memory for other data that needs to be accessed. Especially for memory databases such as Redis, memory is never enough. In addition to splitting the data out to multiple Redis servers. Another way to increase the capacity of a database is to use VMS to swap data that is infrequently accessed on the disk. If our stored data is always accessed with a small portion of the data, most of the data is rarely accessed, and it is true that only a small number of users are often active on the site. When a small amount of data is accessed frequently, the use of VMs not only increases the capacity of a single Redis server database, but also does not affect performance too much.

Redis does not use the virtual memory mechanism provided by the OS but instead implements its own virtual memory mechanism in the user state, which the author explains in his own blog. Http://antirez.com/post/redis-virtual-memory-story.html

The main reason is two.
1.os of virtual memory is a 4k page swapped for the smallest unit. Most of the Redis objects are much smaller than 4k, so there may be multiple Redis objects on an OS page. In addition, the Redis collection object types such as List,set may exist with multiple OS pages. It may eventually cause only 10%key to be accessed frequently, but all OS pages will be considered active by the OS so that the OS will swap pages only if the memory is really exhausted.
2. Exchange Mode compared to the OS. Redis can compress objects that are swapped to disk, and objects saved to disk can remove pointers and object metadata information. Generally compressed objects are 10 times times smaller than objects in memory. This allows Redis VMs to do much less IO operations than OS VMS.

The following are VM-related configurations:

vm-enabled Yes	#开启vm功能
vm-swap-file/tmp/redis.swap	#交换出来的value保存的文件路径/TMP/REDIS.S WAP
vm-max-memory 1000000	#redis使用的最大内存上限, after the upper limit, Redis starts exchanging value to the disk file.
vm-page-size	#每个页面的大小32个字节
vm-pages 134217728	#最多使用在文件中使用多少页面 the size of the interchange File = vm-page-size * vm-pages
vm-max-threads 4	#用于执行value对象换入换出的工作线程数量. 0 means no worker threads are used (described later)

The Redis VMs are designed to ensure that the key is searched, and only the value is swapped into the swap file. So if the memory problem is caused by too many value-small keys, then the VM is not resolved. Like the OS, Redis also swaps objects by page. REDIS specifies that only one object can be saved on the same page. However, an object can be saved on multiple pages. No value is exchanged until the memory used by Redis does not exceed vm-max-memory. When the maximum memory limit is exceeded, Redis chooses older objects. If two objects are as old as the older objects, the exact formula Swappability = Age*log (size_in_memory) is the preferred interchange for the larger object. For vm-page-size settings, you should set the size of the page to fit the size of most objects according to your app. Too big to waste disk space, too small will cause the swap file to be fragmented. For each page in the swap file, Redis corresponds to a 1bit value in memory to record the idle state of the page. So like the number of pages in the configuration above (Vm-pages 134217728) consumes 16M of memory to record the page's idle state. Vm-max-threads represents the number of threads used to swap tasks. If greater than 0 is recommended, set the number of CPU cores to the server. If it is 0, the exchange process takes place on the main thread.

After the parameter configuration is discussed, the following is a brief introduction to how the VM works,

When Vm-max-threads is set to 0 o'clock (Blocking VM)
Swap out
The main thread periodically checks that the memory exceeds the maximum limit, is directly blocked, saves the selected object to the swap file, and frees the memory used by the object, and the process repeats until the following conditions are met
1. Memory usage drops below maximum limit
The 2.swap file is full.
3. Almost all of the objects have been swapped to disk.
Swap in
When there is a client requesting the value to be swapped out of key. The main thread loads the corresponding value object from the file in a blocking manner, which is blocked by the client at the time of loading. Then process the client request

When Vm-max-threads is greater than 0 (threaded VM)
Swap out
When the main thread detects that the use of memory exceeds the maximum limit, the selected object information to be exchanged is placed in a queue for background processing by the worker thread, and the main thread continues to process the client request.
Swap in
If a client request key is swapped out, the main thread first blocks the client that issued the command, and then puts the loaded object's information into a queue for the worker to load. A worker thread notifies the main thread after loading is complete. The main thread then executes the client's command. This method only blocks the client that requested value to be swapped out of key
Overall, the overall performance of blocking VMs is better, as there is no need for thread synchronization, creating threads, and recovering blocked client overhead. But the response is also sacrificed accordingly. The way the threaded VM is the main thread does not block on disk IO, so responsiveness is better. It is recommended to use the blocking VM if our application does not often change in and out and does not care about a bit of delay. For a more detailed introduction to Redis VMS, refer to the links below
Http://antirez.com/post/redis-virtual-memory-story.html
Http://redis.io/topics/internals-vm

Redis Data Summary (ix) virtual memory