Redis persistence Overview

Redis persistence Overview
Redis persistence

Redis provides options for different persistence ranges:

• RDB persistently executes the real-time point (point-in-time) snapshot of the dataset at a specified interval.
• AOF records the write operation received each time on the server. It replays the operation at the server startup to recreate the original dataset. Commands are recorded in append mode in the same format as Redis protocol. When the file size is too large, Redis will re-write the log in the background.
• If you want to, you can completely disable persistence, if you only want your data to exist during the running of the server.
• Both AOF and RDB can be supported on the same instance. Note: In this case, when Redis is restarted, The AOF file will be used to reconstruct the original dataset because it is guaranteed to be the most complete data.

Understanding the advantages and disadvantages of RDB and AOF persistence (trade-offs) is a very important thing. Let's start with RDB:

Advantages of RDB

• RDB is a compact file that represents Redis data at a certain instant point. RDB files are suitable for backup. For example, you may want to archive RDB files for the last 24 hours every hour and save RDB snapshots for the last 30 days every day. This allows you to easily recover data sets of different versions for Disaster Tolerance.
• RDB is ideal for disaster recovery. As a compact single file, RDB can be transferred to a remote data center or Amazon S3 (possibly encrypted ).
• RDB maximizes the performance of Redis, because the only thing that needs to be done during persistence of the Redis parent process is to start (fork) A sub-process to complete all the remaining work by the sub-process. The parent process instance does not need to perform operations such as disk IO.
• RDB is faster than AOF when you restart an instance that saves a large dataset.

Disadvantages of RDB

RDB may be poor when you need to minimize data loss when Redis stops working (such as power outages. You can configure different save points to save the RDB file (for example, after at least five minutes and 100 writes to the dataset, but you can have multiple save points ). However, you usually create an RDB snapshot every five minutes or longer, so once Redis is disabled for any reason, you have to prepare for data loss in the last few minutes.

RDB often calls the fork () sub-process to persist to the disk. If the dataset is large, fork () is time-consuming. The result is that when the dataset is large and the CPU performance is not strong enough, Redis will stop the service client for several milliseconds or even one second. AOF also requires fork (), but you can adjust the frequency of log re-writing without compromising the (trade-off) durability ).

AOF advantages

• Using AOF Redis will be more durable: You can have many different fsync policies: fsync, fsync per second, fsync for each request. Using the default fsync policy per second, the write performance is still very good (fsync is completed by the background thread, and the main thread continues to execute write requests ), even if you only lose one second of Data Writing.
• AOF logs are append files, so they do not need to be located and are not damaged during power failure. Even if, for some reason, the end of the file is a half-written command (the disk is full or otherwise), the redis-check-aof tool can be easily repaired.
• When the AOF file becomes very large, Redis will automatically rewrite it in the background. Rewriting is absolutely safe, because Redis continues to append to the old file, and creates a brand new file using the minimum set of operations required to create the current dataset. Once the second file is created, redis will switch the two files and start appending them to the new files.
• The AOF file contains one operation after another, which is stored in a format that is easy to understand and parse. You can also easily export an AOF file. For example, even if you accidentally and mistakenly use the FLUSHALL command to clear everything, if you do not execute rewrite at this time, you can still save your dataset. You only need to stop the server and delete the last command, restart Redis.

AOF disadvantages

• For the same dataset, the AOF file is usually larger than the equivalent RDB file.
• AOF may be slower than RDB, depending on the exact fsync policy. Generally, if fsync is set to once per second, the performance is still very high. If fsync is disabled, it is as fast as RDB even under high load. However, even with a large write load, RDB can provide a good maximum latency guarantee.
• In the past, we experienced some rare bugs for special commands (for example, blocking commands such as BRPOPLPUSH), which made it impossible to restore data to the Saved state during data loading. These bugs are rare. We also tested them in the test suite, automatically and randomly created complex datasets, and then loaded them to check if everything works,, this type of bug is almost impossible in RDB persistence. To make it clearer, Redis AOF updates an existing status incrementally, just like MySQL or MongoDB, while RDB snapshot creates everything from the beginning again and again, it is more robust in concept. However, 1) Note that every time Redis rewrites AOF, the actual data in the current dataset starts from the beginning, compared with the AOF file that has been appended (or an overwrite to read the old AOF file instead of reading data in the memory), it is more immune to bugs. 2) We have not received a report that the user has detected a crash in the real world.

Who to choose

Generally, you should use both persistence methods to achieve the same data security level as that provided by PostgreSQL.

If you are very concerned about your data, but still can accept data loss from a disaster for several minutes, you can only use RDB separately.

Many users use AOF independently, but we do not encourage this because frequent RDB snapshots are very convenient for database backup, and the startup speed is faster, avoiding the AOF engine bug.

Snapshots

By default, Redis saves the dataset snapshot to the disk, a binary file named dump. rdb. You can set Redis to SAVE the dataset when there are at least M changes to the dataset within N seconds, or you can manually call the SAVE or BGSAVE command.

For example, this configuration will automatically dump the dataset to the disk when Redis has at least 1000 key changes every 60 seconds:

save 60 1000

This policy is called a snapshot.

How to work

Whenever Redis needs to dump a dataset to a disk, the following occurs:

• Redis calls fork (). So we have two processes: parent and child.
• The child process starts to write the dataset into a temporary RDB file.
• When the sub-process completes the new RDB file, replace the old file.

This method can benefit Redis from the copy-on-write mechanism.

Append only files

Snapshots are not very durable ). If your computer running Redis is down, the power cord is disconnected, or you accidentally kill-9 to drop your instance, data recently written into Redis will be lost. Although this is not a big deal for some applications, there are also some scenarios that require full durability, which may not be appropriate in these scenarios.

Appending only files is an alternative and a fully persistent Redis policy. It is available in version 1.1.

You can enable AOF in your configuration file:

appendonly yes

From now on, every time Redis receives the command to modify the dataset, it will be appended to AOF. When you restart Redis, The replays AOF file will be replayed to recreate the status.

Log Rewriting

You can guess that the AOF file will become larger when write operations are continuously executed. For example, if you add a counter 100 times, your data set will only have one key to store the final value, but there are 100 records in AOF. 99 of the records are not required when the current status is rebuilt.

Redis supports an interesting feature: rebuilding AOF in the background without affecting the service client. Whenever you send BGREWRITEAOF, Redis will write a new AOF file that contains the shortest command sequence required to reconstruct the dataset in the current memory. If you are using Redis 2.2 AOF, you need to run the BGREWRITEAOF command from time to time. Redis 2.4 can automatically trigger log rewriting (view the sample configuration file in Redis 2.4 for more information ).

AOF durability

You can configure how long Redis will fsync data to the disk. There are three options:

• Execute fsync every time a new command is appended to the AOF file. Very slow, but very secure.
• Execute fsync per second. Fast enough (2.4 is almost as fast as snapshot), but data will be lost for 1 second in the event of a disaster.
• Never execute fsync and directly submit your data to the operating system. Faster, but less secure.

The recommended (and default) policy is to execute fsync once per second. Fast and secure. The policy that has been executed is very slow in practice (although it has been improved in Redis 2.0), because it cannot make the fsync operation itself faster.

What should I do if AOF is damaged?

It is possible that the server crashes (crash) when writing AOF files, and Redis cannot be loaded after the files are damaged. If this happens, you can use the following steps to solve the problem:

• Create a copy of AOF for backup.
• Use the Redis-check-aof tool that comes with redis to fix the original file:
• $ Redis-check-aof -- fix
• Use diff-u to check the differences between the two files. Restart the server with repaired files.

How to work

Log rewriting adopts the same write-time replication mechanism as snapshot. The process is as follows:

• Redis calls fork (). So we have two processes: parent and child.
• The child process starts to write AOF to a temporary file.
• The parent process accumulates new changes in a memory buffer (and writes New Changes to the old AOF file at the same time, so we are safe even if the rewrite fails ).
• When the child process completes file rewriting, the parent process receives a signal and appends the memory buffer to the end of the file created by the child process.
• Done! Now Redis automatically rename the old file as a new one, and then start to append new data to the new file.

How to switch from RDB to AOF

Redis 2.2 and later versions are very simple and do not need to be restarted.

• Back up your latest dump. rdb file.
• Put the backup file in a safe place.
• Send the following two commands:
• Redis-cli config set appendonly yes
• Redis-cli config set save ""
• Make sure that your database contains the same number of keys in its package.
• Ensure that the write is correctly appended to the AOF file.

The first CONFIG command enables AOF. Redis will block to generate the initial dump file, open the file to prepare for writing, and start the append write operation.

The second CONFIG command is used to disable snapshot persistence. This step is optional if you want to enable both persistence methods at the same time.

Important: Remember to edit your redis. conf file to enable AOF. Otherwise, when you restart the server, your configuration changes will be lost and the server will use the old configuration.

AOF and RDB Interaction

In Redis 2.4 and later versions, AOF rewriting cannot be triggered during RDB snapshot operations, or BGSAVE cannot be run during AOF rewrite operations. This prevents two Redis background processes from simultaneously performing heavy I/O operations on the disk.

When a user uses BGREWRITEAOF to explicitly request log rewriting during snapshot operation, the server will reply an OK status code, telling the user that this operation has been scheduled, when the snapshot is complete, the data will be overwritten.

When both AOF and RDB are enabled, Redis restarts and uses the AOF file to reconstruct the original dataset, because the AOF file is usually the most complete data storage.

Back up data

Make sure to back up your database. Disk damage, cloud instance loss, and so on: No backup means a huge risk of data loss.

Redis is very friendly to data backup, because you can copy the RDB file during database operation: Once the RDB file is generated, it will not be modified, and the file is generated into a temporary file, after a new snapshot is created, the rename (2) Atomic file name is automatically used to change the file name to the target file.

This means that it is completely safe to copy RDB files when the server is running. Our suggestions are as follows:

• Create a scheduled task (cron job), create an RDB snapshot to a directory every hour, and store the snapshots in another directory every day.
• When running a scheduled script, you must use the "find" command to delete the old snapshots. For example, you can save the snapshots of each hour in the last 48 hours and the snapshots of each day in one or two months. Note that the date and time information is added when the snapshot is named.
• At least once a day, your RDB snapshot is transmitted outside your data center, or at least to the physical machine that runs your Redis instance.

Disaster recovery

Disaster recovery in Redis basically refers to backup and transmission of the backup to multiple external data centers. In this way, even if some catastrophic events affect the master data center that runs Redis and generates snapshots, data is also safe.

Because many Redis users are at the startup stage, there is not much money to spend, we will introduce some of the most interesting Disaster Recovery Technologies, instead of spending too much.

• Amazon S3 and some similar services are a good way to help you recover your system from disasters. You only need to transmit your daily or hourly RDB snapshots to S3 encrypted. You can use gpg-c to encrypt your data (in symmetric encryption mode ). Ensure that your password is stored in different security places (for example, to the most important person in your organization ). We recommend that you use multiple storage services to improve data security.
• Use SCP (an SSH component) to transmit your snapshots to a remote server. This is a simple and secure method: Build a small VPS away from you, install ssh, and generate a password-free ssh client key, and add it to the authorized_keys file on your VPS. You can automatically transfer the backup file. To achieve good results, it is best to build at least two VPS from different providers.

It is easy to fail if such a system is not correctly handled. At least make sure that the size of the file is verified after the transfer is completed (to match the file you copied). If you use VPS, you can use the SHA1 digest.

You also need an independent alarm system to trigger an alarm when the transmission backup process is abnormal for some reasons.

You may also like the following articles about Redis. For details, refer:

Install and test Redis in Ubuntu 14.04

Basic configuration of Redis master-slave Replication

Redis cluster details

Install Redis in Ubuntu 12.10 (graphic explanation) + Jedis to connect to Redis

Redis series-installation, deployment, and maintenance

Install Redis in CentOS 6.3

Learning notes on Redis installation and deployment

Redis. conf

Redis details: click here
Redis: click here

This article permanently updates the link address: