Redis persistence Cache

Redis persistence Cache
Redis persistent cache:

I. snapsho (snapshot method-"binary file)

① Periodically generate snapshots

② Generate snapshots quantitatively

Observe the settings of the SNAPSHOTTING setting module in the redis configuration file. We can find that

I have explained the meaning of the save command. Now let me talk about it again, that is:

Save? 900? 1 ????? Triggered when the key value changes at least once every 900 seconds (15 minutes;

?

 

# Whether to block the "change operation" of the client when an error occurs during snapshot and the client cannot continue ", "error" may be caused by the disk being full, disk failure, or OS-level exception. "stop-writes-on-bgsave-error yes"

Rdbcompression?

When the RDB file is too large, it can be compressed. Redis enables compression by default. Of course, you can also disable compression by configuring the rdbcompression parameter.

 

Advantages and disadvantages of compression and non-compression:

Compression:

Advantage: reduces disk storage space. disadvantage: CPU resources are consumed.

Do not compress:

Advantage: no CPU resource consumption disadvantage: large disk space occupation

 

How to choose? You need to check the requirements and server resources.

 

RDB snapshot process:

 

1. redis calls fork and now has sub-process and parent process.

2. The parent process continues to process client requests. The child process is responsible for writing the memory content to the temporary file. Because the OS's copy on write (copy on write) Parent and Child processes share the same physical page, when the parent process processes write requests, the OS creates a copy of the page to be modified by the parent process, instead of writing shared pages. Therefore, the data in the subprocess address space is a snapshot of the entire database at the fork moment.

3. After the sub-process writes the snapshot to the temporary file, it replaces the original snapshot file with the temporary file, and then the sub-process exits. The client can also use the save or bgsave command to notify redis to perform snapshot persistence. The save operation saves snapshots in the main thread. Because redis uses a main thread to process all client requests, this method will block all client requests. Therefore, it is not recommended. Note that each snapshot persistence completely writes the memory data to the disk, instead of synchronizing and changing the data incrementally. If the data volume is large and there are many write operations, a large number of disk I/O operations will inevitably occur, which may seriously affect the performance.

 

 

Manual snapshot:

 

If Automatic snapshots are not triggered, you can perform manual snapshot operations on redis. Both SAVE and BGSAVE can execute manual snapshots. The difference between the two commands is that the former is a snapshot operation by the main process, other requests will be blocked, while the latter takes snapshots through the fork sub-process.

 

Note:

Because redis uses fork to copy the current process, the sub-process will occupy the same memory resources as the main process, for example, the main process 8 GB memory, therefore, you must ensure 16 GB of memory during backup. Otherwise, the virtual memory will be enabled, and the performance will be very poor.

2. append only mode (aof is similar to a prewrite log)

① Background execution

② Backup at the service side

 

Redis AOF is similar to the log mechanism. Every write operation is written to the hard disk. When the system crashes, AOF can be used to restore data. Every command with write operations is recorded on the AOF file when the Redis server receives the command. Because it is only an append operation to the file, the operation to write to the hard disk is usually very fast.

In fact, the Redis oaf mechanism involves two things: rewrite and AOF. Rewrite is similar to the log recovery point of the common database management system. When the AOF file expands with the running of the write command, the rewrite will be run when the file size reaches the critical point.
Like replication, rewrite generates a sub-process fork, creates a temporary file, traverses the database, and outputs each key and value pair to a temporary file. The output format is the Redis command. To reduce the file size, multiple key and value pairs are assembled and expressed using one command. Write operations during rewrite are stored in the rewrite buffer in the memory. After successful rewrite, these operations are also copied to the temporary file, and the temporary file will replace the AOF file.

If AOF is disabled when AOF is enabled, the rewrite operation can be performed using the bgrewriteaof command.

 

The Redis Bgrewriteaof command is used to asynchronously execute an AOF (AppendOnly File) File rewriting operation. Rewriting creates a volume optimized version of the current AOF file.

Even if Bgrewriteaof fails, no data is lost because the old AOF file is not modified before Bgrewriteaof is successful.

Note:Starting from Redis 2.4, AOF rewrite is automatically triggered by Redis. BGREWRITEAOF is only used to manually trigger the rewrite operation.

 

 

Automatic bgrewriteaof

To prevent the aof file from being too large, bgrewriteaof will be made periodically to restructure the aof file. In the past, we used to configure crontab to execute this command in off-peak hours. The addition of a workaroud script task is very bad in large clusters and is not easy to check. errors cannot be detected in real time.

Therefore, this automatic bgrewriteaof function is directly added to the redis internal. First, for the aof file, add a field to the server object to record the size of the aof file server. appendonly_current_size. This field is maintained every time aof changes.

After bgrewriteaof is completed or the instance starts to load aof data, it will also call the aofUpdateCurrentSize function to maintain this field, and record the size of the aof file server at this time. auto_aofrewrite_base_size is used as the reference value to determine the aof growth rate.

With the current value and reference value, we can determine the growth of aof files. You also need to configure two parameters to determine whether to automatically trigger bgrewriteaof.

? Auto-aof-rewrite-percentage: bgrewriteaof is triggered when the size of the aof file exceeds the baseline percentage. The default value is 100, which means bgrewriteaof is triggered when the current aof is twice the benchmark size. Set it to 0 to disable the auto-trigger function.
Auto-aof-rewrite-min-size: the number of bytes of the current aof file to be triggered. Avoid unnecessary behavior when aof is small. The default size is 64 mb.
Both parameters can be statically configured in the conf file or dynamically modified using the config set file.

Appendonly? Whether to enable the aof Cache Mechanism

Appendfilename ??? Aof file name

Appendfsync ?? Three execution methods:

 

The fsync () method is called to allow the operating system to write data to the disk. There are three modes of data synchronization:

Always ????????? Every call, for example, security, but the slowest speed;

Everysec ?????? Synchronization per second, which is also the default method;

No? ? ???????? If fsync is not called, the operating system determines when to synchronize data, such as the quick mode;

No-appendfsync-on-rewrite ?? No

 

As mentioned above, the bgrewriteaof mechanism is used to rewrite aof in a sub-process, so that the main process does not block the processing of other commands, and the aof file is too large. Now the problem occurs. When you perform the bgrewriteaof operation and write the aof file to the master process, both operations operate on the disk, while bgrewriteaof usually involves a large number of disk operations, this will cause the main process to block When writing the aof file. Now the no-appendfsync-on-rewrite parameter is available. If this parameter is set to no, It is the safest way to avoid data loss, but the blocking problem must be tolerated. What if it is set to yes? This is equivalent to setting appendfsync to no, which means that the disk operation is not performed, but the buffer is written, so this will not cause blocking (because there is no competing disk ), however, if redis fails at this time, data will be lost. How much data is lost? Under the default settings of linux, data of up to 30 s is lost.

Therefore, if the application system cannot tolerate latency but can tolerate a small amount of data loss, set it to yes. If the application system cannot tolerate data loss, set it to no.

Iii. Summary:

AOF and RDB have their own advantages and disadvantages, which are determined by their respective characteristics:

1) AOF is more secure and can synchronize data to files in a more timely manner. However, AOF requires a lot of disk I/O expenses. The size of AOF files is large, and the file content recovery is relatively slow.
* 2) snapshot provides poor security. It is the best way to back up data in the "normal period" and synchronize data from the master-slave. The file size is small and the recovery speed is fast.

You can specify one of them through the configuration file, or use them at the same time (it is not recommended to use them at the same time), or disable all of them. In a well-structured environment, the master usually uses AOF, the main reason why slave uses snapshot is that the master must first ensure data integrity, which is the first choice for data backup. slave provides read-only services (currently, slave can only provide read services ), its main purpose is to quickly respond to client read requests. However, if your redis runs in poor network stability or poor physical environment, we recommend that you use AOF for both master and slave, this reduces the time cost of "manual data backup"/"manual data recovery" when switching the master and slave roles. If your environment is very good, the service needs to receive intensive write operations, so it is recommended that the master take snapshot, and the slave adopt AOF.

Redis data recovery

When the redis server fails, the data will be restored to the memory according to the following priority during restart:

1 .? If only AOF is configured, load the AOF file to restore data upon restart.

2 .? If both RBD and AOF are configured, only the AOF file is loaded at startup to restore data.

3 .? If only RDB is configured, the dump file is loaded at startup to restore data.