Preface redis is often used for persistent storage in the project, but I don't know the principle of redis persistence. I will repost a blog here, so I can take a book note. Original article address: Workshop:
- The client sends write operations to the server (the data is in the memory of the client)
- The database server receives the write request data (the data is in the memory of the server)
- The server calls write (2) This System Call and writes data to the disk (the data is in the system memory buffer)
- The operating system transfers the data in the buffer zone to the disk controller (the data is cached on the disk)
- The disk controller writes data to the physical media of the disk (the data actually falls on the disk)
Fault Analysis and write operations generally involve the above five processes. Next we will take a look at various levels of Faults Based on the above five processes.
- When the database system fails, the system kernel is still OK at this time, so as long as we finish step 1, data is secure, because the subsequent operating system will complete the next steps, ensure that the data will eventually be stored on the disk
- When the system is powered off, all the caches mentioned in the preceding five items will expire and the database and operating system will stop working. Therefore, only after Step 1 is completed can the machine be powered off to ensure that the data is not lost, and the data in the above four steps will be lost.
Through the above five steps, we may want to find out the following problems:
- How long does the database call a write (2) and write data to the kernel buffer?
- How long does the kernel write data in the system buffer to the disk controller?
- When will the disk controller write the cached data to the physical media?
For the first problem, we usually have full control over the database. For the second problem, the operating system has its default policy, but we can also use POSIX
The fsync commands provided by API force the operating system to write data from the kernel to the disk controller. For the third problem, it seems that the database is no longer accessible, but in fact, in most cases, the disk cache
Is set to disabled, or is only enabled as read cache, that is, write operations are not cached and directly written to the disk. We recommend that you enable the write cache only when your disk has a backup battery. Data corruption means that data cannot be recovered. We have discussed how to ensure that data is indeed written to a disk, but writing data to a disk does not mean that data will not be damaged. For example, we may
Two different write operations are performed for one write request. In case of an accident, one write operation may be completed safely, but the other is not performed yet. If the structure of the data file in the database is unreasonable
The data is completely irrecoverable. There are usually three policies to organize data to prevent data files from being damaged to unrecoverable:
- The most rough processing means that data can be recovered without being organized. Instead, you can configure the data synchronization backup mode to recover the damaged data file through data backup.
- The other is to add an Operation Log Based on the above, and remember the operation behavior each time, so that we can recover data through the operation log. Because operation logs are written in sequence append mode, operation logs cannot be restored.
- It is more secure that the database does not modify old data, but only writes data in append mode. In this way, the data itself is a log, in this way, data cannot be recovered.
Next, let's talk about the first persistence policy of redis, RDB snapshot. Redis supports the persistence mechanism of saving the snapshot of the current data into a data file. The number of continuous writes
How can I generate a snapshot for a database? Redis uses the fork command copy on
Write mechanism. When a snapshot is generated, the current process fork out of a sub-process, and then loop all the data in the sub-process, to write data into an RDB file, we can use the Save command of redis to configure the RDB snapshot generation time. For example, you can configure to generate a snapshot when there are 100 writes within 10 minutes, it can also be configured as 1 small
A snapshot is generated when 1000 write operations are performed. Multiple rules can be implemented together. These rules are defined in the redis configuration file. You can use the redis config
The SET command sets rules when redis is running, and the RDB file of redisredis does not need to be restarted, because the write operation is performed in a new process. When a new RDB file is generated, the child process generated by redis first writes data
Temporary files, and then rename the temporary files to RDB files by calling the atomic rename system. In this way, redis RDB files are always available in any time when a fault occurs. At the same time, the redis RDB file is also a part of the internal implementation of redis master-slave synchronization. However, we can clearly see that RDB has its shortcomings, that is, once there is a database problem, the data stored in our RDB file is not brand new, from the last RDB file generated
All data generated during the downtime of apsaradb for redis is lost. In some businesses, this is tolerable. We also recommend that these businesses use RDB for persistence, because the cost of enabling RDB is not
High. However, for other applications that require extremely high data security and cannot tolerate data loss, RDB is powerless. Therefore, redis introduces another important persistence mechanism: aof day.
Log. Aof log aof log is append only
File, we can see from the name that it is an append log file. Unlike the BINLOG of a general database, the aof file is identifiable plain text and its content is one by one
Redis Standard Commands. For example, we conduct the following experiment and set the aof function in the startup command parameters:
[HTML]View plaincopy
- Sudo sed-I "s/appendonly no/appendonly Yes/g"/etc/redis. conf & sudo/etc/init. d/redis-server restart
Then run the following command: view the aof log file and you will get the following information: The write operation generates a corresponding command as the log. It is worth noting that the last del command is not recorded in the aof log because redis judges
This command does not modify the current dataset. Therefore, you do not need to record this useless write command. In addition, aof logs do not generate logs completely based on client requests, such as commands
The incrbyfloat log is recorded as a set record, because floating point operations may be different on different systems, so to avoid the same log from generating different data on different systems
So here we only record the aof rewrite using the set result after the operation. You may think that every command generates a log. Will the aof file be large? The answer is yes. The aof file will become larger and larger, so redis provides another function,
Aof
Rewrite. The function is to regenerate an aof file. Only one operation is performed on the new aof file, unlike an old file, operations on the same value may be recorded multiple times. Its
The generation process is similar to RDB. It is also a fork process that directly traverses data and writes data to a new aof temporary file. During the process of writing new files, all the write operation logs will still write the old aof
Files are also recorded in the memory buffer. After the operation is complete, all logs in the buffer zone are written to the temporary file at one time. Call the atomic rename name and use the new aof file to retrieve
From the above process, we can see that the RDB and aof operations are sequential Io operations with high performance. At the same time, when the database is restored through the RDB file or aof log, the data is also read sequentially and loaded into the memory, so it will not cause random disk read. What is the purpose of importing persistent data? Of course, it is used to recover data after restart. Redis is a memory database. Both RDB and aof are measures to ensure data recovery. Institute
When redis uses RDB and aof for recovery, it will read the RDB or aof file and reload it To the memory. Compared with the start time of MySQL and other databases,
Because MySQL does not need to load data to the memory. However, when MySQL is started to provide services, the accessed hot data will be slowly loaded into the memory, which is usually called push, and before the push is complete, its performance will not be too high. The advantage of redis is to load data to the memory at one time and push data at one time. In this way, as long as redis is started, the service delivery speed is very fast. However, there are some differences in the startup time between RDB and Aof. The RDB startup time is shorter, because there are two reasons. Each data record in an RDB file has only one record, no
As with aof logs, there may be multiple operation records of one piece of data. Therefore, you only need to write each data entry once. Another reason is the storage format of RDB files and the encoding of redis data in memory.
The format is consistent and data encoding is not required. The CPU consumption is much smaller than that of aof logs.
