Wasp: Exploring the distributed database of massive data in Alibaba

 

Author: Liu Xuhui Raymond reprinted. Please indicate the source

Email: colorant at 163.com

Blog: http://blog.csdn.net/colorant/

=
What is=

 

Wasp is an hbase-based database solution developed by Alibaba Group. Its fundamental starting point is to follow Google's External Store, "hbase systems provide cross-row transactions, indexing, and SQL functions without sacrificing linear expansion capabilities"

 

=
Architecture Principle=

 

For the design principle, refer to the related thesis of external store. You can find the wasp related design documentation in the following two places.

 

Https://github.com/alibaba/wasp/wiki/Chinese

Http://wenku.baidu.com/view/c85f50d984254b35eefd345c.html

 

The core idea of the External Store framework is to divide the data into different entitygroups. The data backup of entitygroup is stored across DataCenter and provides complete Acid support within the entitygroup, ensure that data write operations are synchronized and backed up in all data centers.

 

 

 

From the specific implementation point of view, wasp does not implement external store in cross-Data
The related design ideas of the center only use solutions such as entity groups to divide and manage data.

 

In many designs, External Store focuses on the core idea of ultra-large-scale data concurrency, such as entity groups's cross-region backup, when reading data, non-master-slave equal nodes are dynamically selected by paxos. They are all designed to ensure decentralization during read operations to improve performance, the wasp architecture scheme is more like hbase's scheme. There are fmaster nodes and fserver nodes, and the current fmaster is determined through zookeeper. Each fserver manages several entity
Groups is basically a fixed master-slave center. In the use of entity group, Wasp basically retains the original design of External Store, through redolog
/MVCC/cross-entity two-phase commit and other methods to solve the consistency problem of concurrent read/write

 

=
Implementation=

 

Wasp uses Alibaba's DRUID project to parse SQL syntax, and uses netty and protobuf to construct the server's Internal Communication Protocol framework.

 

Wasp data is mainly mapped to four types of tables on hbase, global
_ Fmeta _ the table records the meta information of all wasp tables. Each wasp table corresponds to an entity table with the same entitygroup.
The redolog table and index table corresponding to all tables under the key jurisdiction.

 

Currently, Wasp supports SQL syntax in a simple way. For example, query only supports equal.
Condition and the Compare class range condition on the index. The support for int and other data structures also has bugs in comparison operations. Other SQL syntaxes that are slightly more complex, such as UDF, limit,
Having, group by, join, order by, and so on are not available at present. Of course, this may also depend on the specific application scenarios of wasp. You may only need the simplest equal and the range of a specific field.
Condition class query.

 

In addition, from the perspective of SQL plan implementation, it seems that hbase operations such as get, put, and delete are simply converted. From the perspective of hbase, hbase is a pure client application, no hbase is used
RS-side capabilities, such as filter and coprocessor, are optimized. Therefore, to implement the aggregation class function, the performance may be greatly affected.

 

=
Summary=

 

In general, Wasp cannot provide a massive data cross-Data Center solution, and its scale is subject to a single hbase
To a certain extent, there is still a big gap between cluster and objective issues solved by external store. Wasp provides more enhanced solutions on hbase, provides simple SQL interfaces and support for cross-row transactions. If the light from sqlon
From the hbase perspective, there is a big gap with saleforce's Phoenix. However, it is better than Phoenix in cross-row transaction support.
(Phoenix's transaction support is almost entirely dependent on hbase's own capabilities.) Currently, code functions are not yet mature, but it depends on future development. Of course, from the perspective of the code framework and design pattern, the Programming Skill of the author is still very good and should be learned.

 

I just quickly learned about the implementation of wasp, and its own capabilities are limited, so the accuracy of the above views is not guaranteed. please correct me if there is any deviation.