The persistence of Redis

Redis is an advanced Key-value database. It is similar to memcached, but the data can be persisted and the supported data types are rich. There are strings, lists, collections, and ordered collections. Supports the addition, intersection and complement (difference) of the compute sets on the server side, and supports a variety of sorting functions. So Redis can also be viewed as a data structure server.
All redis data is stored in memory and then periodically asynchronously saved to disk (this is called "semi-persistent mode"), and each data change can be written to a append only file (AOF) (this is called "Full persistence mode").

The first method filesnapshotting: The default redis is to persist data to disk in the form of a snapshot (a binary file, Dump.rdb, this file name can be specified), the format in the configuration file is: Save N m indicates within n seconds , Redis takes snapshots to disk if at least m modifications occur. Of course we can also manually perform save or Bgsave (asynchronous) to take snapshots.

How it works: when Redis needs to be persisted, Redis will fork a child process, the child process writes the data to a temporary RDB file on disk, and when the child process finishes writing the temporary file, the original RDB is replaced, so the advantage is that you can Copy-on-write

Another way to persist is append-only: The Filesnapshotting method is lost when Redis is dead (the amount of data lost depends on the configuration of your save policy), so this is its biggest disadvantage, when the volume of business is large, There is a lot of lost data. The Append-only method can do all the data without loss, but the performance of Redis will be worse. AOF will be able to achieve full-length persistence, only need to open in the configuration file (default is no), appendonly Yes on aof, Redis every time you execute a modified data command, will add it to the AoF file, when the Redis restart, will read the aof file to "replay "To revert to the last moment before Redis was closed.

Log rewriting as the execution of the modified data aof file becomes larger, many of which record the change of a key. So Redis has a more interesting feature: rebuilding the aof file in the background without affecting the client side operation. Executing the bgrewriteaof command at any time will write commands for the shortest sequence in the current memory to disk, which can completely build the current data situation without any extraneous changes (such as state changes, counter changes, etc.) and the size of the aof file. so when using AOF, Redis recommends using bgrewriteaof at the same time.

AoF file Refresh method, there are three, reference configuration parameters Appendfsync :appendfsync always each commit a modification command calls Fsync flush to the aof file, very very slow, but also very safe; appendfsync everysec calls Fsync refresh to aof file every second, soon, but may lose less than one second of data;Appendfsync no relies on OS refresh, Redis does not actively refresh aof, This is the quickest, but the security is poor. The default and recommended refresh per second, so that both speed and security are done.

AOF may be damaged due to system reasons, Redis can no longer load this aof, you may follow the steps below to fix: First make a aof file backup, copy to other places; repair the original aof file, execute:$ redis-check-aof–fix; You can use the Diff–u command to see where files are inconsistent before and after the repair, and restart the Redis service.

LOG rewrite works: The same applies to copy-on-write: First Redis will fork a child process, the child process writes the latest aof to a temporary file, and the parent process increments the most recent execution in memory to write (at this point, the old AoF is still written. Rewrite if the failure is also secure); When the child process finishes rewrite the temporary file, the parent process receives a signal and writes the previous in-memory incremental changes to the end of the temporary file, where Redis renames the old aof file, renames the temporary file, and begins writing to the new aof.

Finally, in case (the machine is broken or the disk is broken), remember to regularly back up the *rdb *.aof files that were generated using filesnapshotting or append-only to the remote machine. I was using crontab every half an hour to SCP once. I did not use Redis's master-slave function, because half-hour backup should be possible, and I think there is a waste of machine if the decision. This eventually depends on the application.

========================

Data persistence is a popular way to save data to disk and ensure that data is not lost due to power outages and other factors.

Redis needs to frequently synchronize the in-memory data to disk to ensure persistence. Redis supports two persistence modes, one is snapshotting (snapshot) is the default, and the other is the way Append-only file (abbreviated AOF). First introduce the two dump ways to talk about some of their own phenomena and ideas, the front of the content is collated from the Internet.

Snapshotting
Snapshots are the default persistence mode. This is the way in which the in-memory data is written to the binary in a snapshot, the default file name is Dump.rdb. You can automatically make snapshot persistence by configuring settings. We can configure Redis to take snapshots automatically if more than M key is modified in n seconds, the following is the default snapshot save configuration

Save 1 #If more than 1 key is modified in 900 seconds to initiate a snapshot save
Save #300 seconds of content is modified if more than 10 keys, the snapshot save is initiated
Save 60 10000

The detailed snapshot save procedure is described below

1.redis calls fork and now has child and parent processes.

2. The parent process continues to process the client request, which is responsible for writing the memory contents to the temporary file. Because the OS's write-time replication mechanism (copy on write) will share the same physical page, when the parent process processes the write request, the OS creates a copy of the page to be modified by the parent process, rather than writing the shared page. So the data in the child process's address space is a snapshot of the entire database at fork time.

3. When the child process finishes writing the snapshot to the temporary file, replace the original snapshot file with a temporary file, and the child process exits.

The client can also use the Save or Bgsave command to notify Redis to do a snapshot persistence. The save action is to save the snapshot in the main thread, which blocks all client requests because Redis uses a main thread to process all client requests. So it is not recommended. It is also important to note that each snapshot persistence is a full write of the memory data to disk once, not the incremental synchronization of only dirty data. If the amount of data is large, and more write operations, it will inevitably cause a large number of disk IO operations, may seriously affect performance.

In addition, because the snapshot is done at a certain interval, if Redis is accidentally down, all changes after the last snapshot will be lost. You can use AOF persistence if your application requires that you cannot lose any modifications. Described below

Append-only file

AOF is more persistent than snapshot mode, because Redis appends each received write command to a file by using the Write function (default is appendonly.aof) when aof persistence is used. When Redis restarts, the contents of the entire database are rebuilt in memory by re-executing the write commands saved in the file. Of course, because the OS caches write modifications in the kernel, it may not be written to disk immediately. The persistence of this aof method is also likely to lose some of the modifications. But we can tell Redis through the configuration file that we want to force the OS to write to disk through the Fsync function. There are three ways to do this (the default is: Fsync once per second)

AppendOnly Yes//enable AOF persistence mode
# Appendfsync always//write command is immediately forced to write to disk, the slowest, but guaranteed full persistence, not recommended
Appendfsync everysec//force write disk once per second, a good compromise in performance and persistence, recommended
# Appendfsync no//completely dependent on OS, best performance, no guarantee of persistence

The AOF approach also poses another problem. Persistent files can become more and more large. For example, we call the INCR Test command 100 times, the file must save all 100 commands, in fact, 99 are redundant. Because you want to restore the state of the database, it is enough to save a set test 100 in the file. In order to compress the aof persistence file. Redis provides the bgrewriteaof command. Receive this command Redis will use a snapshot-like method to save the in-memory data to a temporary file in the form of a command, and finally replace the original file. The specific process is as follows

1. Redis Call Fork, now has a parent-child two processes
2. The child process writes to the temporary file the command to rebuild the database state based on the database snapshot in memory
3. The parent process continues to process the client request, in addition to writing the write command to the original aof file. Cache the received write commands at the same time. This will ensure that if the child process rewrite fails, it will not be problematic.
4. When a child process writes the snapshot content to a temporary file, the child process signals the parent process. The parent process then writes the cached write command to the temporary file as well.
5. Now the parent process can replace the old aof file with the temporary file and rename it, and the subsequent write commands are also started to append to the new aof file.

Note that the operation to rewrite the aof file does not read the old aof file, but instead overwrites the entire in-memory database content with a new aof file in the form of a command, which is a bit similar to a snapshot.

the idea on operation and maintenance

In fact, the snapshot and aof, like the use of copy-on-write technology. Several experiments found that each time the data dump, the memory will be enlarged by one times (about this problem can be referred to the Redis memory trap I wrote last year, many people use Redis as a cache, the amount of data is small, dump time is very short, so it is not easy to find), this time there will be three kinds of situation:

A: physical memory enough to meet, this time dump very fast, the best performance

Second: Physical memory + virtual memory can be satisfied, this time the dump speed will be slow, disk swap busy, service performance will also fall. Fortunately, after a long time, the data dump was completed, and then the memory returned to normal. The system is poorly stable.

Three: Physical memory + virtual memory can not meet, this time the dump has been dead, time is long the machine hangs. This situation is a disaster!

If the data is to be persisted and the stability is to be guaranteed, half of the physical memory is suggested to be left blank. If you feel unacceptable or have a way, here are the following:

Snapshots and aof use Copy-on-write, but with a different point, snapshots you can't predict when Redis will do dump,aof can control the dump by bgrewriteaof command.

Based on this I can open multiple Redis nodes on one server (using multi-CPU), using AOF persistence mode.

For example, in the 24G memory of the server to open 3 nodes, daily with bgrewriteaof to reorganize the data periodically, each node dump time is different, so theoretically each node can consume 6G of memory, a total of 18G memory, and another 6 g in the presence of a single node dump, Memory, use 6g!. Of course, the more nodes open the more memory utilization is also higher. If bandwidth is not an issue, the number of nodes is recommended = number of CPUs.

In order to ensure high performance in my application, the data did not dump or use AOF. Because the failure to do dump is much lower than when the dump, even if the data is lost, the automatic repair script can immediately recover data. After all, the massive data Redis can only do data fragmentation, then the amount of data falling on each node is not much.

Redis Virtual Memory recommendations also do not use, with Redis is to achieve the abnormal performance, virtual memory, AOF looks a little chicken.

Now can not be separated from Redis, because its mget is now all the best performance in the DB, previously also considered using Tokyocabinet hash method to do mget, performance does not give force. Directly with Redis, basically a single redis node mget can reach 10w/s

Error Correction

When I said that Redis was doing data dump, the content would be enlarged a bit, and then I did some tests and found that some places were wrong.

The top command is not a reflection of the actual memory footprint, although the fork-out subprocess occupies the same amount of memory as the parent process, but when dump does not have a write operation, it is actually using the same memory data. When there is a write operation of the memory will be real expansion (is not the actual expansion of one-time uncertainty, the data may be partitioned according to the page), this is the real copy-on-write.

Based on this, it is more flexible to do data persistence.

The persistence of Redis