Redis learning Manual (Virtual Memory)

I. Introduction:

Like most nosql databases, redis also follows the key/value data storage model. In some cases, redis will save keys/values in the memory to improve the efficiency of data query and data modification. However, this is not always a good choice. In view of this, we can further optimize it, that is, to keep only keys data in the memory as much as possible, so as to ensure the efficiency of data retrieval, however, when values data is rarely used, it can be swapped out to the disk.
In actual applications, only about 10% of keys belong to relatively common keys, so redis can replace other infrequently used keys and values to the disk through virtual storage, once the exchanged keys or values needs to be read, redis will read it back to the main memory.

II. Application scenarios:

For most databases, the most ideal operation mode is to load all the data into the memory, and the subsequent query operations can be completed completely based on the memory data. However, in reality, such scenarios are not common. In more cases, only some data can be loaded into the memory.
In redis, there is a very important concept, that is, keys are generally not exchanged, so if your database has a large number of keys, each key is associated with only a small value, therefore, this scenario is not very suitable for using virtual memory. If, on the contrary, the database only contains a small number of keys, and the value associated with each key is very large, this scenario is no longer suitable for the use of virtual storage.
In practical applications, in order to make the virtual storage more fully available to help us improve the system operation efficiency, we can merge keys with many smaller values into keys with a small number of smaller values. The most important method is to change the original key/value mode to the hash-based mode, so that many original keys can become attributes in the hash.

Iii. Configuration:

1) Add the following configuration items in the configuration file to enable the VM function when the current redis server is started.
VM-enabled Yes

2) set the maximum number of available virtual memory bytes for redis in the configuration file. If the data in the memory is greater than this value, some objects are swapped out to the disk. The memory occupied by the swapped out object is released until the used memory is less than this value.
VM-max-memory (bytes)
Redis's switching rule is to consider the "Oldest" data whenever possible, that is, the data that is not used for the longest time will be swapped out. If the age of the two objects is the same, the data with a large value will be swapped out first. It should be noted that redis will not switch keys to the disk. Therefore, if only the data of keys has been filled with the entire virtual storage, this data model will not be suitable for using the virtual storage mechanism, or set the value to a greater value to accommodate the entire keys data. In practical applications, if we want to use redis virtual memory, we should allocate as much memory as possible for redis to avoid frequent exchanges.

3) in the configuration file, set the number of pages and the number of bytes occupied by each page. To transfer data in the memory to the disk, we need to use swap files. These files are not related to data persistence. redis will delete them all before exiting. Because most of the access methods to swap files are random access, it is recommended to store swap files on Solid State disks, which can greatly improve the operating efficiency of the system.
VM-pages 134217728
VM-page-size 32
In the preceding configuration, redis divides the swap file into VM-pages Pages, where each page occupies VM-page-size bytes, the swap file size available for redis is: VM-pages * VM-page-size. A value can be stored on one or more pages, but a page cannot hold multiple values, we need to fully consider this feature when setting VM-page-size.

4). There is a very important configuration parameter in the redis configuration file, namely:
VM-max-threads 4
This parameter indicates the maximum number of threads that redis uses when performing Io operations on the swap file. Generally, we recommend that you set this value to the CPU cores of the host. If this value is set to 0, redis performs this operation in synchronous mode when performing Io interaction with the swap file.
For redis, if the exchange file is operated in synchronous mode, when a client is accessing the data in the exchange file, if other clients attempt to access the data in the exchange file, the client request will be suspended until the previous operation is completed. Especially when reading large data values on a relatively slow or busy disk, the impact of this blocking is even more abrupt. However, synchronous operations are not useless. In fact, from the perspective of global execution efficiency, the synchronous mode is better than the asynchronous mode. After all, the synchronous mode saves thread switching and inter-thread synchronization, and additional overhead of operations such as thread pulling. Especially when most frequently used data can be directly read from the main memory, the synchronization mode will be superior.
If your real application is the opposite, there will be a lot of switch-in and switch-out operations, and your system has a lot of cores, in view of this, you don't want the client to have to block a short period of time before accessing the swap file. If so, I think the asynchronous method may be more suitable for your system.
As for the final configuration method, the best answer will come from continuous experiments and tuning.