Laxcus Big Data Management System 2.0 (13)-Summary

Summary

The main components and applications of Laxcus are expounded from several angles. All designs are based on real-world assessment, comparison, testing and consideration. The basic idea of the design is very clear, that is, the functions of decomposition, refinement, classification, the formation of one can be independent, small modules, each module to undertake a function, and then organize these modules, in a loose coupling framework management, cooperation, to complete large-scale data storage and computing work.

The main problem in design stems from the contradiction between limited infrastructure and changing application requirements. How to maximize the utilization of limited infrastructure resources without loss of processing performance is the focus of design considerations. This is also a hardware-related issue.

We face a variety of options when approving system design objectives. In most cases, there is no chance of having a fish or a cake, only two choices. Considering that although the system has undergone many simplification, but in order to meet the needs, there are still too many links, coupled with the complexity of the various linkages between these links, we will be stable, reliable, "big" the three indicators in the first place, other indicators as the secondary requirements are put behind. So, from this point, Laxcus, although able to manage millions's computer nodes, realizes the EB-level data storage computing power, but also provides a fast memory-based data processing solution, but not a big data system designed for the pursuit of "fast".

Now go back, according to our experience, to see a cluster of our organization implemented and put into operation of the basic configuration of the situation: a computer from hundreds of to thousands of computers, using the X86 architecture of the 32/64-bit chip, each computer Configuration 2-8g memory, 2-4TB reserves of Winchest hard disk. Computers are deployed on multiple racks, with gigabit fiber networks that connect through multiple switches. During different periods, there will be hundreds of to tens of thousands of users, each of which uses the cluster for multiple tasks concurrently.

You can see that in such a network environment, each computer can be assigned to a small bandwidth, considering that Laxcus is a multiuser system, each user in parallel to perform multiple tasks, the actual allocation of bandwidth for each task is lower. Cluster in the execution of normal data processing business, there are a large number of ancillary services to transmit data through the network, if the total data usage statistics, such bandwidth is prone to overloading phenomenon, and cluster operation is also heavily dependent on network communications, this part of the bandwidth resources can not be squeezed. Therefore, how to save the various storage and computing overhead in the process of data processing becomes very critical.

One way to save is to eliminate redundant data.

Redundant data is mainly generated in the data generation phase, the prevention means is very simple: the data to accurately filter and crawl. This involves the design of the storage scheme.

To accommodate different business needs, we have designed a row/column two storage model. The minimum storage unit for a storage model is a column. At the column level, the data can be organized, replaced, and calculated at will, while guaranteeing flexibility without sacrificing precision. Associated with this, the integration of SQL into the distribution description language, combined with the distribution environment, has many effects. For example, with the use of the multi-conditional combination query capability of the WHERE clause, which is dispersed to the data access plane by a SELECT statement, the results of the search as a unit can be generated directly on the computer, thus avoiding redundant data transmission over the network. This data processing scheme is very consistent with the principle of computing the location of data sources in a distributed environment. That is, the principle of mobile computing instead of mobile data computing.

The savings principle is also reflected in the original definition of the data format.

Laxcus design rules, the values are uniformly used in binary format. For example, an integer value, the binary format is fixed to 4 bytes, if the string is expressed in addition to the symbol bit, the maximum will reach 11 bytes. This allows binary values to be stored in both disk and memory, or on the network, with less data than in other formats. And because the length of the numerical value is fixed, when the CPU level is processed, it is not necessary to convert the data type, it can be recognized and calculated by the CPU, such as C language can be directly referenced by pointers. This can significantly improve computational efficiency when performing data-intensive computing tasks.

The FIXP agreement is another example of a binary advantage. In network communication, the UDP-based monitoring packet is a large proportion, because the binary makes the FIXP packet data length is relatively small, usually less than the length of an IP packet, also lower than the link layer of the packet size limit. The result is that the FIXP packet can be transmitted directly, which avoids the grouping and reorganization of packets at both ends of the network. This small improvement makes the packet loss rate of the FIXP protocol greatly reduced, the unreliable UDP communication success rate greatly increases, obtains the higher communication stability.

Another consideration in the design is the problem of data computing under the network environment.

Data blocks are the key to solving these problems.

As described in Chapter 3rd, the basic feature of a data block is the fixed length. This alone avoids disk fragmentation and reduces the difficulty of data maintenance. For this reason, it is very simple to spread the data between networks, which lays a foundation for the data backup and load balance in the network environment. At the same time, the disk most affected by the write operation to the memory execution, in a space-time approach, the disk write operation delay to a minimum, so that the disk more focused on the read operation. Further, by introducing data optimization and data structuring, users are more able to organize and retrieve data as they wish. Performance on disk, the direct reflection is to reduce the number of read operations. These are important to improve the performance of disk processing and improve the efficiency of data calculation.

Outside of the data block, the most important thing to mention is the Diffuse/converge algorithm, which is the core of the entire network computation. The algorithm divides the network computing process into two steps: diffuse to find the data on the network after the allocation, converge on the basis of the allocated data, the data re-organization and redistribution, the output of each data as the next input, through a number of iterations to obtain the results of the calculation. Around the diffuse/converge algorithm, and a series of supporting design. By separating the functions of the system and the user, the user realizes the network computing interface can be derived programming, the system realizes the hot release, the task scheduling, the model average allocation data, the task naming, the conduct statement, causes these modules to carry on each part, the runtime organizes, forms from the terminal to the data storage stratification plane, A complete network computing system. While maintaining simplicity and ease of use, it also achieves the purpose of parallel computing of large scale data.

Redundant disaster management is also included in the Network Computing system. Because of the ability of weak center management and fault active judgment, the system can perceive the existence of fault in a short time, and actively avoid the source of fault. For the fault node, once the system is determined, it will be isolated, no longer in the cluster, and the use of data redundancy and re-recovery methods to protect the normal data processing business.

In addition, in many technical details, also optimized processing. For example, in the implementation of the underlying calculation, for the data-dense feature, the addition of the SSE command X86 architecture, can make the computational efficiency exponentially increased. Or in the calculation of the use of more addition, subtraction, shift instructions, to replace the multiplication, except the instruction, but also to reduce the instruction calculation period, improve computational efficiency. There is also a more direct means of using 64-bit CPU, according to the test statistics, compiled by the Linux GCC64-bit compiler of C + + language code, in the execution of intensive data calculations, efficiency than 32-bit code generally increased 15%-20%.

For the future, we should choose suitable hardware and software combination according to the specific characteristics of cluster and data processing business. As the article begins, Cluster computing is not the strength of the individual computing Unit's performance, it is to organize a large number of decentralized computing units through the network to work together in a way that replaces those single but powerful centralized computing. This is a multi-win feature, to achieve low-cost computing business to bring the gospel. In this case, the data center is particularly visible. In today's data centers, computers and refrigeration devices that maintain their computers consume very large amounts of power. If the use of mobile architecture hardware equipment, not only the computer energy consumption will be significantly reduced, the computer volume is reduced, the number of computers in the unit space increased, the demand for refrigeration will be greatly reduced. When such an infrastructure is deployed to the data center, it will directly reduce the operational costs of the data. Its impact, just as the PC architecture replaces the same year's minicomputer, now mobile architecture to replace the PC architecture has shown the trend. Today's emphasis on total cost of ownership (TCO), coupled with the advent of big data, will unleash a new revolution in data computing.

This may not be a lengthy process, and there is much work to be done.

Now it's just the beginning.

Laxcus Big Data Management System 2.0 (13)-Summary