Technical paper: A cloud storage reliability mechanism oriented to erasure code technology

A cloud storage reliability mechanism based on Erasure code technology

Du Yunyin

In cloud storage, data can be divided into three types: frequent reads, frequent writes, and cold data. Different kinds of data are set up in different encoding mechanism: The multiple copy technique is used for frequent reading data, the single file coding scheme is used to write the data frequently, and the cold data adopts the cross file coding scheme. In the cloud storage system, this paper presents a new method for the reliability of the erasure code, which is easy to implement, has high fault tolerance and can reduce the resource consumption compared with the multiple copy mechanism. Based on a RAID packet on HDFs, the prototype of Rubik's Cube code is implemented, and the Blaumroth and RDP are added to compare the test. Although each kind of coding has the advantage and disadvantage, through the single machine and the cluster test RAID-5, Blaumroth, RDP and Rubik's cube code the compile code speed, the Rubik's cube code decoding speed is higher than the existing optimal RAID-6 code RDP code to raise one order of magnitude. At the same time, a three-dimensional erasure code is proposed for a large scale distributed system, which can tolerate any 7 errors. In short, the test results show that Rubik's cube code compared to three copies of the technology, in reducing the consumption of half of the resources at the same time can guarantee high availability and reliability.

A cloud storage reliability mechanism based on Erasure code technology