Redis: 8. Master-slave replication and virtual memory

Redis master-slave replication is easy to configure and use. Master-slave replication allows multiple slave servers to have the same database copies as the master server. The following are some features of redis master-slave replication:

1. The master can have multiple slave instances.

2. In addition to connecting multiple slave instances to the same master, slave can also be connected to other slave instances to form a graph structure.

3. Master-slave replication does not block the master. That is to say, when one or more slave and master synchronize data for the first time, the master can continue to process the requests sent by the client. On the contrary, slave blocks requests that cannot process the client when synchronizing data for the first time.

4. Master-slave replication can be used to improve the scalability of the system. We can use multiple slave for client read requests. For example, sort operations can be processed using slave. It can also be used for simple data redundancy.

5. You can disable data persistence on the master. You only need to comment out all the Save configurations in the master configuration file, and then configure data persistence only on the slave.

The following describes the process of master-slave replication.

After the slave server is configured, slave establishes a connection with the master and sends the sync command. The master will start a background process to save database snapshots to files, whether it is the first synchronization established connection or the re-connection after the connection is disconnected, at the same time, the master process starts to collect new write commands and cache them. After the background process completes file writing, the master sends the file to slave. slave saves the file to the disk and then loads it To the memory to restore the database snapshot to slave. Then the master will forward the cache command to slave. In addition, the write commands received by the master will be sent to slave through the established connection. Command for synchronizing data from Master to slave and
The commands sent by the client use the same protocol format. When the master and slave are disconnected, slave can automatically establish a new connection. If the master receives synchronous connection commands from multiple slave instances at the same time, only one process is started to write the database image and then sent to all slave instances.

It is easy to configure the slave server. You only need to add the following configuration in the configuration file:

Slaveof 192.168.1.1 6379 # specify the master's IP address and port

Virtual Memory:

First, we will explain that redis's virtual memory and OS's virtual memory are not the same thing, but their ideas and purposes are the same. It is to temporarily swap infrequently accessed data from the memory to the disk, so as to free up valuable memory space for other data to be accessed. Especially for memory databases such as redis, the memory is always insufficient. In addition to splitting data into multiple redis servers. Another way to increase database capacity is to use VM to swap infrequently accessed data disks. If a small portion of the stored data is frequently accessed and most of the data is rarely accessed, the website is always active only a small number of users. When a small amount of data is frequently accessed, using VM not only improves the performance of a single redis instance
Server database capacity, and does not have too much impact on performance.

Redis does not use the virtual memory mechanism provided by OS, but implements its own virtual memory mechanism in user mode. The author explains the reason in his blog. Http://antirez.com/post/redis-virtual-memory-story.html

There are two main reasons:

1. The virtual memory of the OS is switched in the smallest unit of 4 K pages. Most redis objects are much smaller than 4 K, so there may be multiple redis objects on an OS page. In addition, the set object type of redis, such as list, may exist on multiple OS pages. In the end, only 10% keys may be frequently accessed, but all OS pages will be regarded as active by the OS, so that the OS will switch pages only when the memory is actually exhausted.

2. Compared with OS. Redis can compress the objects exchanged to the disk. Objects saved to the disk can remove pointer and object metadata information. Generally, the compressed object is 10 times smaller than the object in the memory. In this way, redis VMS can perform much less Io operations than OS VMS.

The following are VM configurations.

VM-enabled yes # enable the VM Function

VM-Swap-file/tmp/redis. Swap # path of the file stored by the exchanged value/tmp/redis. Swap

VM-max-memory 1000000 # maximum memory used by redis. After the upper limit is exceeded, redis starts to exchange values to disk files.

VM-page-size 32 # the size of each page is 32 bytes.

VM-pages 134217728 # maximum number of pages used in files, swap file size = VM-page-size * VM-pages

VM-max-threads 4 # Number of worker threads used to execute value object switching. 0 indicates no working thread is used (described later)

Redis VM is designed to ensure the key search speed, only the value is exchanged to the swap file. Therefore, if the memory problem is caused by too many keys with small values, the VM cannot solve the problem. Like OS, redis also exchanges objects by page. Redis requires that only one object can be saved on the same page. However, an object can be saved on multiple pages. No value is exchanged before the memory used by redis exceeds the VM-max-memory. When the maximum memory limit is exceeded, redis selects older objects. If the two objects are the same, the larger objects will be exchanged first. The precise formula is swappability.
= Age * log (size_in_memory ). For VM-page-size settings, set the page size to the size that can accommodate most objects according to your application. If the disk space is too large, the disk space will be wasted. If the disk space is too small, the swap file will be broken. For each page in the swap file, redis will correspond to a 1-bit value in the memory to record the idle status of the page. As shown in the preceding configuration, the number of pages (Vm-pages 134217728) occupies 16 MB of memory to record the idle page status. VM-max-threads indicates the number of threads used for switching tasks. If it is greater than 0, we recommend that you set the number of CPU cores to the server. If the value is 0, the switching process is performed in the main thread.

After the parameter configuration is discussed, let's briefly introduce how the VM works,
When VM-max-threads is set to 0 (blocking VM)

Swap out

When the main thread regularly checks and finds that the maximum memory is exceeded, the selected object is saved to the swap file and the memory occupied by the object is released, this process repeats until the following conditions are met

1. The maximum memory usage is as follows:

2. The swap file is full.

3. Almost all objects are switched to the disk.

Inbound

When a client request value is exchanged for a key. The main thread will load the corresponding value object from the file in blocking mode, and the client will be blocked during loading. Then process the client request
When VM-max-threads is greater than 0 (threaded VM)

Swap out

When the main thread detects that the memory usage exceeds the maximum limit, it puts the selected object information to be exchanged in a queue for processing by the working thread background, and the main thread continues to process client requests.

Inbound

If the key requested by a client is swapped out, the main thread first blocks the client that issues the command, and then places the information of the loaded object in a queue for the worker thread to load. The worker thread notifies the main thread after loading. The main thread then executes the client command. This method only blocks the client whose request value is exchanged with the key.

In general, the blocking VM method has better performance, because thread synchronization is not required, and thread creation and recovery are congested. However, responsiveness is also sacrificed. Threaded VM mode the main thread does not block the disk Io, so it is more responsive. We recommend using blocking VM if our applications do not frequently change in and out, and do not care much about latency. For details about redis Vm, refer to the following link.
Http://antirez.com/post/redis-virtual-memory-story.html
Http://redis.io/topics/internals-vm

From: http://www.cnblogs.com/xhan/archive/2011/02/07/1949717.html

Redcreen