Seven-cow storage algorithm guessing

Catalogue [-]

• Introduction: What is the erasure code
• The base domain ——— finite field of operation in erasure code
• Seven-cow storage solution

When you browse the Web, you have a habit of tagging. The recent collation of past labels, found that accumulated a number of seven cattle company storage strategy pages, and decided to organize an article processing, in case of memory. Of course, also want to be useful to others.

Because seven cattle Company's storage strategy is mainly based on the Erasure code (Erasure codes,ec), so the following first from the Erasure code extension.

Introduction: What is the erasure code

The explosion of data has increased the size of storage systems and the reliability of storage devices has not been significantly improved (SSD from SLC to MLC and TLC reliability is declining, the disk with the unit area to write data more resulting in increased reliability), so that the data of persistent storage poses a huge challenge. In addition, as the size of storage system increases, the increase of cold data in storage system will be far more than the increase of thermal data, how to save cold data safely, and the ability to acquire cold data when needed is an important challenge in storage system.

Replica policy and coding strategy are two important methods to ensure data redundancy. When the raw data is partially lost, both the copy policy and the encoding policy ensure that the data is still properly fetched. The copy policy stores the original data in one or more copies, and the encoding strategy blocks and encodes the original data to generate redundant blocks of data, guaranteeing that the original data is still available.

The two policy performance comparisons are as follows:

|| Strategy | | Storage Efficiency | | calculation Overhead | | Repair Efficiency | |

|| replica Policy | | Low | | No | | High | |

|| Coding Strategies | | High | | have | | Low | |

Although the coding strategy is less computationally expensive and less efficient than the replica policy (described later), the benefits of significantly reducing storage overhead are a huge space for coding strategies. In practice, storage systems with replica policies and coding policies, such as thermal data in distributed storage systems (HDFS, GFS, TFS) are often saved through a copy policy, and cold data is saved by encoding, saving storage space. ECC encoding policy is also used to ensure data redundancy in ECC memory and SSD page.

From the point of view of information theory and coding, erasure code belongs to packet linear coding. The coding process can be expressed by a coding matrix GM and the multiplication (dot product) of the block data, which means that the coding matrix GM defines how the data is encoded as redundant data.

C0 ~ C5 is redundant data, all redundant data can be represented as multiplication of gmxd{d0, D1, D2, D3}, each redundant data block CI is the corresponding line of the matrix and the product of the data block (yellow mark). Each element of GM in the coding matrix is the multiplication factor corresponding to the original data block.

Above excerpt from: Erasure Codes in storage systems (Erasure Codes)-Overview (i)

The base domain ——— finite field of operation in erasure code

• What is a finite field?

Mathematically, the construction of some symbols, and their operations based on them, constitutes a "domain". such as integer numbers and the thought operations above them (subtraction) Form an integer field. If the number of symbols in a field is limited, it is called a finite field, otherwise an infinite field. For example, an integer field is an infinite field, and a Boolean field that we are familiar with is a finite field. The size of a finite field is the exponential power (p^q) of prime numbers (recorded as p) or primes.

• The operational properties of a domain
  • Addition and multiplication are closed, addition and multiplication results are also within the domain
  • The commutative law of addition and multiplication; a + b = B + A and A. B = B. A
  • Assignment law of addition and multiplication: A + (b + c) = (A + b) + C and A. (b c) = (a b) · C
  • Assignment law of Addition and multiplication: A. (b + c) = (a B) + (a C)
  • Unit elements with addition and multiplication: a+0 = A and a 1 = 1 (here The 0,1 is not a natural number 0, 1, which represents the addition unit element and multiplication unit element on the field)
  • The elements on each domain have their negative and inverse entities: A + (-a) = 0, A. (a-1) = 1
• Computer operation and Finite field

As you know every computer language, whether static or dynamic language, has its most basic data types, each data type has its data range, which determines that the operation must be a finite field operation. The finite field was first proposed by the mathematician Galois, a finite domain construction method even if it is proposed by him, also known as Galois field, such a domain as GF (p) and GF (PQ), the computer's P is 2, because its data is binary form, and its operation is based on binary operations, namely GF (2w) , W can be 1, 2, 4, 8, 16 and so on. If W is 1, then the addition operation of GF (2) is an XOR operation, and the multiplication is the operation. When W is greater than 1 o'clock, its addition operation is still not different or budgeted, and the multiplication operation is a slight difference, which is no longer detailed here.

Seven-cow storage solution

• "Pioneer" Seven Qiniu storage: three years grinding sword and behind the core technology analysis

The core of the technology storage system is the cost and reliability, and these two are a contradiction, want to reduce the risk of loss of data, it is bound to increase the number of copies of each copy of data, and increase the number of copies of each copy of the data, but also bound to increase the cost, seven of cows using the EC redundancy algorithm to balance the contradiction. This algorithm divides a piece of data into m parts, and the M data into a multivariate linear equation group, calculates the N checksum data, then stores the m+n data, in the stored m+n data, any one or more data corruption, we can through this multivariate linear equation group to calculate the damaged data back, By this principle we can easily get a conclusion that the system using the EC redundancy algorithm supports up to n data corruption without loss of data, seven KN uses some of its own unique technology, the M and n are relatively large values, M is far greater than N, so that the system's backup number is very low-(m+n)/m (m much larger than N), And the reliability is very high-can simultaneously damage N-part data (n is also a large number). I've talked about seven cows. Using EC redundancy algorithm to obtain a very high data reliability, and on this basis, the introduction of dual data center interoperability to avoid a single room catastrophic accident, through these efforts, seven of cattle to protect the enterprise data 0 lost.

• Xu Xiwei: 11 years upstream downstream, chief architect to CEO

The domain algebra follows the subtraction law of the natural algebra, but the data values are controlled in a finite area, regardless of how it is calculated, the results are in the domain of 0 to 255, so called domain algebra. The storage file can be considered a sequence of 0 to 255, for example, a 100K file is split into 10 parts, each 10K, there are 10 places, but the file is still a copy. At this time with the addition of domain algebra (in fact, the computer's XOR or operation), from the 10 data to take out a calibration data, the data into 11 copies, its redundancy is 1.1. This is a checksum-based storage method, the cost is low, but the effect and the double copy almost, any one of the data is lost, can be restored back.

• «What's on the storage System-seven cow new storage (V2) on-line Xu Shiwei on June 5, 2014»

The first highlight of the new storage is the introduction of an arithmetic redundancy scheme such as the Erasure Code (EC), which is no longer a classic 3 copy redundancy scheme. Our EC uses 28+4, which divides the file into 28 parts, then calculates 4 redundant data based on the 28 data, and finally stores the 32 data on 32 different computers. The advantage of this is that it is both inexpensive and improves reliability and usability. From a cost perspective, the same is to store 1PB of data, to buy a storage server as long as 3 copies of the storage of 36.5%, the economic benefits are quite good. In terms of reliability, the previous 3 copies allowed only colleagues to damage 2 disks, and now allows 4 disks to be damaged at the same time, which intuitively greatly improves reliability. From a usability standpoint, it was possible to accept 2 servers before they were offline, and now allow 4 servers to be offline at one time. The second largest two points for new storage is the repair speed, and we've lifted the single-disk repair time from three hours to less than 30 minutes. Repair time is of great importance to improve reliability. The 28+4 (m=4) scheme guarantees a data loss probability of 1e-16, which means 16 9 reliability can be achieved.

The above paragraph to be discussed is, through 36.5% of this economic value is not know how Mr Xu is calculated? I think it's 38.95% "(28+4)/(28 * 3) = 38.95%".

The above three introductory paragraphs are a few words from the seven-cow storage scheme I collected from the Internet. It can be confirmed that the calculation process must be based on the finite-field erasure code algorithm. Cannot confirm is its Erasure code algorithm, namely 28+4 (m=4) plan "If according to seven Cow Cloud storage CTO Han To, should be m=28, n=4", how is this 4 redundant data calculated?

Generally speaking, the algorithm of erasure code is divided into two categories, XOR code and RS Code, the XOR code is based on the finite field GF (2), because its codec addition operation based on the computer can quickly complete the XOR operation (Bit-wise exclusive-or). The common XOR codes are:

• Low density parity check codes (Density Parity code, LDPC)
• Cauchy-Reed Solomon Code (Cauchy-reed-solomon Codes,crs)
• RAID codes (e.g. RAID5, RAID6)
• Parity code (EVENODD)
• X-Size (X-code)

From the second paragraph introduction, it can be confirmed that seven cattle storage algorithm should be based on GF (2^3) finite domain, its erasure code algorithm should be an XOR code algorithm. Its GM should be 4 rows and 7 columns matrix (each element of the matrix can also be regarded as a matrix block, that is, a data block), D should be 7 rows of a column matrix, the check matrix C should be four rows of a column matrix (each element is a checksum data block).

Individual projections can only end there. Perhaps with more information to be disclosed by the company after seven, I will continue to supplement this article.

The above analysis may be a fallacy, but also invites experts to throw bricks, continue to revise this article, in order to make common progress.

Seven-cow storage algorithm guessing