Redis Persistence and master-slave replication (RPM)

Author: silenceper

Date: 2013-10-03

Original address: http://silenceper.com/archives/959.html

One of the biggest differences between Redis and memcached is the persistence of Redis support data, the ability to keep the in-memory data on disk, and the ability to reload for use when restarting. I. The persistence provided by Redis:

1). RDB Persistence:
The mechanism is to write a snapshot of the in-memory data set to disk within a specified interval of time.
2). AoF Persistence:
This mechanism will record every write processed by the server as a log, and read the file in the early of the Redis server to rebuild the database to ensure that the data in the database is complete after startup.
3). No persistence:
We can disable the persistence of the Redis server by configuring it so that we can treat Redis as a feature-enhanced version of memcached.
4). Apply both AoF and RDB.

Ii. advantages and disadvantages of the RDB mechanism:

What are the advantages of an RDB?
1). Once this is used, your entire Redis database will contain only one file, which is perfect for file backups. For example, you might want to archive the last 24 hours of data every hour and archive the last 30 days of data every day. With such a backup strategy, we can recover very easily once the system has a catastrophic failure.
2). The RDB is a great choice for disaster recovery. Because we can easily compress a single file and then transfer it to other storage media.
3). Maximum performance. For a Redis service process, the only thing it needs to do when it starts to persist is to fork out the child process and then complete the persistence work by sub-processes, which can greatly prevent the service process from performing IO operations.
4). If the data set is large, the RDB will be more efficient to start than the AOF mechanism.

What are the disadvantages of an RDB?
1). An RDB is not a good choice if you want to ensure high availability of data, which minimizes data loss. Since the system is down before the scheduled persistence, the data that has not been written to the disk before is lost.
2). Since the RDB assists with data persistence through the fork process, it can cause the entire server to stop serving hundreds of milliseconds, or even 1 seconds, if the data set is large.

Third, the advantages and disadvantages of the AOF mechanism:

What are the advantages of aof?
1). This mechanism can result in higher data security, i.e. data persistence. There are 3 synchronization policies available in Redis, that is, synchronization per second, synchronization per modification, and unsynchronized. In fact, synchronization per second is also done asynchronously, and its efficiency is very high, the difference is that once the system is down, then the data will be lost in a second. Each time we modify the synchronization, we can treat it as a synchronous persistence, that is, each occurrence of the data changes will be immediately logged to disk. It can be predicted that this is the least efficient way. As for the no-sync, needless to say, I think we can understand it correctly.
2). Because this mechanism writes to log files in append mode, it does not break the content that already exists in the log file even if there is an outage during the write process. However, if we only write half of the data this time there is a system crash problem, do not worry, before the next boot of Redis, we can use the Redis-check-aof tool to help us solve the problem of data consistency.
3). If the log is too large, Redis can automatically enable the rewrite mechanism.

(trigger mechanism of AOF rewrite

If Redis only stores instructions for client-side modification of the database in the AoF file, the size of the AoF file will continue to increase because the aof file simply reproduces the instructions that stored the client and does not merge. The simplest way to deal with this problem is that when the aof file satisfies certain conditions, the AOF is rewrite,rewrite according to the data in the current memory database to traverse to write to a temporary aof file, and then replace the original aof file after writing.

There are three types of Redis trigger aof rewrite mechanisms:

1. Redis server receives the BGREWRITEAOF instruction request sent by the client, if the current AOF/RDB data persistence is not executing, then executes, and vice versa, and so on the current AOF/RDB data persistence after the end of execution aof rewrite

2. In Redis profile redis.conf, the user sets the Auto-aof-rewrite-percentage and Auto-aof-rewrite-min-size parameters, and the current aof file size Server.aof_ Current_size greater than auto-aof-rewrite-min-size (Server.aof_rewrite_min_size), AoF Rewrite is automatically triggered when the growth rate of aof file size is greater than auto-aof-rewrite-percentage (SERVER.AOF_REWRITE_PERC)

3, the user Set "config set appendonly yes" when the aof is turned on, call startappendonly function will trigger rewrite)
4). AOF contains a well-formed, easy-to-understand log file for recording all modification operations. In fact, we can also complete the reconstruction of the data through this file.

What are the disadvantages of aof?
1). aof files are typically larger than the Rdb file for the same number of datasets.
2). Depending on the synchronization strategy, the AOF is often slower than the RDB in terms of operational efficiency. In summary, the efficiency of the synchronization policy per second is high, and the efficiency of the synchronization disable policy is as efficient as the RDB. Iv. Persistent Configuration

1. Snapshotting:
By default, Redis dumps a snapshot of the dataset to the Dump.rdb file. In addition, we can modify the frequency of the Redis server dump snapshot through the configuration file, after we open the 6379.conf file, we search for save, we can see the following configuration information:
Save 1 #在900秒 (15 minutes), dump memory Snapshot if at least 1 key changes occur.
After save #在300秒 (5 minutes), dump memory Snapshot if at least 10 key changes occur.
After save 10000 #在60秒 (1 minutes), dump memory Snapshot if at least 10,000 key changes occur.

2. Dump snapshot mechanism:
1). Redis first fork the child process.
2). The child process writes the snapshot data to the temporary RDB file.
3). After the child process completes the data write operation, replace the old file with the temporary file.

3. aof File:
It has been said many times that the RDB snapshot timing dump mechanism does not guarantee good data persistence. If our application is really concerned about this point, we can consider using the AOF mechanism in Redis. The default mechanism for Redis servers is the RDB, and if you need to use AOF, you need to modify the following entries in the configuration file:
Change AppendOnly No to appendonly Yes
From now on, Redis appends the data modification commands to the AoF file every time it receives a command. The next time Redis restarts, the information in the AoF file needs to be loaded to build the most up-to-date data into memory.

4. Configuration of AOF:
There are three synchronization modes in the Redis configuration file, namely:
Appendfsync always #每次有数据修改发生时都会写入AOF文件.
Appendfsync everysec #每秒钟同步一次, this policy is the default policy for AOF.
Appendfsync no #从不同步. Efficient but data is not persisted.

5. How to repair aof files for bad damage:
1). Make an additional copy of the existing AoF file that has been damaged.
2). Execute the "redis-check-aof--fix <filename>" command to repair the damaged AoF file.
3). Restart the Redis server with the repaired aof file.

6. Redis Data Backup:
In Redis, we can back up the running Redis data files online by copy. This is because the Rdb file will no longer be modified once it is generated. Each time Redis dumps the latest data into a temporary file, it then renames the temporary file to its original data file name using the Rename function atomicity. So we can say that copy data files are safe and consistent at any time. For this reason, we can periodically back up Redis data files by creating cron jobs and copy the backup files to secure disk media. v. Realization of master-slave replication Master/slave

Configuration from the server:

Locate the REPLICATION zone in the configuration file and join:

Slaveof 192.168.1.103 6379//slaveif Primary server IP Home server port

RDB persistence is turned on by default

To enable AOF persistence from the server

[Shell]
Dir./  //default
appendonly Yes
appendfilename appendonly.aof
[/shell]

The primary server configuration takes the default rules.

This allows the data on master to be copied to the slave.

Other operations:

1. Use redis-cli bgsave command to persist data on Master Redis once a day and CP to other backup servers.
2, with REDIS-CLI bgrewriteaof command every half hour to persist the data on the slave Redis, and CP to other backup servers.
3, write a script, regular get master and slave on the key, see if two is synchronized, if there is no synchronization, timely alarm.

Test

Adding data to Master

[Shell]
Redis 192.168.1.103:6379> set name Silenceper
OK
[/shell]

You can use get name to get the data on slave.

[Shell]
Redis 192.168.1.104:6379> Get name
"Silenceper"
[/shell]

Use Phpredisadmin to see changes to your data more intuitively. Six, the principle of master-slave replication

Whether it is a first-time connection or a reconnection, a SYNC command is sent from the server to the primary server when a slave server is established.

The master server that receives the SYNC command will begin executing BGSAVE and will save all newly executed write commands to a buffer during the save operation execution.

When the BGSAVE finishes executing, the master server sends the. rdb file that the save operation sends to the slave server, receives the. rdb file from the server, and loads the data in the file into memory.

The master server then sends all the content accumulated in the Write command buffer to the slave server in the format of the Redis command protocol.

You can personally verify the synchronization process by using the Telnet command: first connect to the Redis server that is processing the command request, and then send it the Sync command, and after a while you will see that the Telnet session (session) received a large segment of data from the server (. Rdb file), and then you'll see that all the write commands that were executed by the server are resent to the Telnet session.

Even if multiple slave servers are sending sync to the primary server, the primary server can handle all of these synchronization requests from the server by simply executing the BGSAVE command once.

From the server can be automatically re-connected when the connection between the master and slave server is disconnected, before the Redis 2.8 version, the reconnection from the server will always perform a full resynchronization operation, but starting from the Redis 2.8 version, from the server can Select whether to perform a full resynchronization or partial resynchronization (partial resynchronization) based on the primary server.