Dedup Technology Introduction 1

De-duplication is a very important concept in the storage field, especially in the Data Backup field. The purpose is to delete duplicate data blocks to reduce the use of buckets.

The starting point of this idea is very natural. Generally, some of the data backed up at a time always overlaps with the data backed up at a time.

For example, each backup must contain a MB file, and the file will appear in all the backup data again.

After multiple backup operations, duplicate data blocks occupy considerable storage space. In fact, it is sufficient to retain one copy of these duplicate data blocks.

Dedup is generated to solve this problem.

Dedup and data compression are two different concepts.

Both of them can reduce storage space usage, which is what they have in common.

Data Compression requires a large amount of CPU and IO resources, and can be used only after decompression. In addition, data compression cannot solve the problem of duplicate data blocks.

Dedup also requires a large amount of CPU resources, but greatly reduces the occupation of Io resources.

Since duplicate data blocks need to be eliminated, the first problem for dedup is how to divide data blocks.

The simplest and easiest way to think of it is to divide it with a fixed length. For example, each data block is 1 MB.

This method is easy to implement and efficient.

The disadvantage is also obvious, that is, the effect on adding/deleting data is not obvious.

For example, a 10 MB file is divided into 10 data blocks during the first backup.

Then, a character is added at the beginning of the file, and the file length is changed to 10 MB + 1 byte. In the next backup, the file will be divided into 11 data segments. The first 10 data segments are 1 MB, and the last data segment contains 1 byte.

However, these 11 data segments are regarded as new data segments, so they do not achieve the expected results of dedup.

There are many ways to solve this problem. The easiest way to think of it is to find out whether it is repeated with existing data blocks in the new backup data based on different offset values.

This will solve the data deduplication problem, but it will also bring a lot of CPU consumption. A common additional solution is scroll checksum, which is used to reduce the CPU overhead caused by the checksum of the calculated data block.

Of course, there are many better solutions to this problem, but this method is widely used because it is simple and easy to implement. For example, the famous rsync tool is implemented in this way.

This article from the "drib" blog, please be sure to keep this http://nilei.blog.51cto.com/9133015/1551697

Dedup Technology Introduction 1