Do you know the Write Hole problem in RAID?

Do you know the Write Hole problem in RAID?

In the RAID system, there will be a problem called Write Hole. What is the problem? What is the impact on users? How can we solve this problem in the open-source RAID system? How does one solve these problems? From the perspective of R & D, storage Engineer Wu analyzed the problem and looked at its nature and solutions.

Write Hole is a very vivid description. For a system that uses the internal verification data for data protection, when the application is updating the internal data of the Strip, if the system suddenly loses power, the strip data cannot be updated normally, in this case, the data in the strip is partially updated. In this case, after the system is restarted, the data in the strip is incomplete, the validation data cannot match the valid data in the Strip, and the strip is inconsistent with the data. This issue is called the Write Hole issue, as shown in:

When the Strip data is updated, the D2 data is updated to D2, but the validation data P and Q of the strip are not updated. In this case, the system restarts due to a fault. In this case, the strip data is inconsistent. After the system is restarted, the RAID system cannot perceive the data inconsistency. In this case, if D3 in the Strip encounters another fault, you must verify the data P to restore D3. When the D2 data is used during data recovery, it is clear that the recovered data will be wrong, and it is no longer the original D3 data. Therefore, the data inconsistency strip will further introduce the data correctness problem. That is, the Write Hole problem not only damages the updated data, but also may affect the correctness of the unupdated data. In extreme cases, the correctness of the entire data with the data is affected.

When the Write Hole problem occurs, that is, when the band data is updated, the system suddenly loses power, and the RAID system contains several inconsistent data bands. After the system is restarted, you can use a full system scan to repair the inconsistent data strip. This is also a common method used by many RAID systems. This method can prevent inconsistent data bands from further leading to the correctness of unupdated data. SYNC is usually used in RAID systems. After the system is restarted illegally, you can use the SYNC function to check all the strip in RAID and fix the inconsistent data strip. This method can prevent the spread of the Write Hole problem, but it still causes the complete loss of some data in the RAID system.

To completely solve the Write Hole problem, you can use the following two methods when designing RAID:

1. Use the Journal (log) design idea of the file system to implement Atomic processing of write requests

2. Use non-volatile memory as the write cache to achieve atomic write operations

These two ideas are essentially similar, and the second is the optimization of the first one. We know that when the file system updates data, in order to ensure the atomicity of data updates, we can use the LOG method to regard a data update operation as a transaction (transaction ). That is, no matter how many I/O operations are performed in a data update operation, it is an atomic operation externally, which can be called atomic write. When the intermediate process of atomic write fails, you can continue the operation after the system restarts to ensure that the data is either updated or completely not written, and there is no intermediate state. To solve the Write Hole problem, the RAID system also needs to introduce this Atomic write mechanism. The direct idea is to record the log, save the event type and related data each time the RAID updates the Strip data, and then submit the data to the Strip; after all the Strip data is updated, clear the log data. This method ensures atomic writing. Even if a system fault occurs during the Strip data update process, after the system is restarted, you can continue the write process through the log data to ensure that the strip data is completely updated, and is in the Data Consistency status.

The log method seems perfect, but it seriously affects the performance. In a disk storage system, log updates are time-consuming. Multiple disk IO operations are required for one update. If the log storage space and the strip storage space are physically discontinuous, the disk jitter problem will be introduced, seriously affecting the performance. For the flash storage system, after the introduction of the log system, all data operations will introduce double data write operations, and the write amplification problem becomes very tricky. Therefore, this log operation method is not provided in the open-source RAID system. The upper-layer file system is usually required to solve the write hole problem. In some commercial RAID systems, to ensure the ultimate data reliability, some systems have implemented this log data update method to achieve Atomic Write.

Another better way than the LOG method is to introduce non-Easy memory NVRAM. When updating RAID data, the data is first written into non-Easy Loss memory. After data aggregation in NVRAM is complete, the data is updated and then written to the Strip. After the data in the strip is updated, clear the data in NVRAM. During the process of updating the striped data, if the system restarts abnormally, the data in the striped data will be updated again because there are still unfinished data in NVRAM, there will be no intermediate data status and no Write Hole issue.

Using NVRAM is a perfect solution to Write Hole. For hard RAID, Hot Standby batteries or super capacitors are often used to protect the memory on RAID cards. Therefore, hard RAID is superior to soft RAID in this respect. In Soft RAID, PCIe NVRAM or NVDIMM can be used in the system to implement non-Easy loss data caching for RAID. Generally, in a storage system, soft RAID works with the file system and the upper-layer application system. Therefore, NVRAM shares with the file system to solve the Write Hole problem. Many EMC storage systems use non-easy-to-lose memory NVRAM, so that file systems can work with RAID to prevent RAID from being in the Write Hole state.

Write Hole is a hard-to-Solve Problem of RAID. However, with the help of the latest NVDIMM and other non-Easy memory loss technologies, this type of problem can be effectively avoided and solved.

How to build a RAID 10 array on Linux

Debian soft RAID Installation notes-use mdadm to install RAID1

Common RAID technology introduction and demo (Multi-chart)

The most common disk array in Linux-RAID 5

RAID0 + 1 and RAID5 Performance Test Results

Getting started with Linux: disk array (RAID)

This article permanently updates the link address: