Migration practice from relational MySQL to NoSQL hbase

November 2013 22-23rd, as the only large-scale industry event dedicated to the sharing of Hadoop technology and applications, the 2013 Hadoop China Technology Summit (Chinese Hadoop Summit 2013) was held at four points by Sheraton Beijing Group Hotel. Near-thousand CIOs, CTO, architects, IT managers, consultants, engineers, and Hadoop enthusiasts from various http://www.aliyun.com/zixun/aggregation/17611.html "> Industry Fields" It vendors and technologists working on Hadoop research and promotion will be involved.

In Sql&nosql Special, senior engineer from Millet Liu Shaohui did "from relational MySQL to NoSQL hbase migration practice" speech. The HBase, as well as the hbase in millet, the important characteristics of millet development and the future development plan are introduced in detail.

First, Liu Shaohui in the introduction of HBase, he said, millet from the beginning of last year's research and testing, has more than 10 business access to the HBase, including the full storage of rice chat messages, micloud SMS call record synchronization, millet push service, as well as some off-line data analysis business.

Liu Shaohui that the typical hbase cluster contains 3 to 5 control nodes, 5 to n data nodes. The main difference between the control node and the data node on disk, the control node's capacity requirements for the disk is not very high, mainly requires the disk to be stable.

▲ Typical hbase cluster

Liu Shaohui continue to introduce, the entire system Automation deployment is millet own development Minos tool, directly in the command line operation, can be very simple to create a cluster, modify the configuration file, upgrade the cluster, delete the cluster functions. At the same time, Minos also integrates a monitoring function. Currently, the tool has been open source in the GitHub, millet internal use version and open source version is basically consistent (Https://github.com/xiaomi/Minos).

Introduction to best practices, Liu Shaohui said, "We encapsulate a layer of hbase client on the basis of native HBase client." This is mainly to achieve four things, one is to ensure the safety of the thread, the second is in the HBase client can automatically add performance indicators, the third is across the table, across the cluster operation for the user is consistent, four can do dynamic update client configuration. ”

For some important business, we use dual-master replication, and realize automatic switching of primary standby cluster. When we confirm the need to switch the cluster, as long as the simple modification of a configuration, the new read and write requests will be switched to the standby cluster.

▲ Best Practices

In addition, there are some other optimizations, such as smooth upgrades, which reduce the unavailable time using the move region script; Full GC, triggering full GC at low peak every day; Compaction, improve the performance of the writing; four is shortcircuit Read; security, Kerberos authentication.

In the end, talking about future development plans, Liu Shaohui introduced, we want to do synchronous replication, across the row across the table atomicity, global level Two index, as well as do some compaction optimization, failover related optimization, multi-tenant sharing cluster with a common cloud, hmaster refactoring. In the area of common development with the community, Liu Shaohui continued, we will also participate in hbase feedback back to the community, and will follow the community's latest progress, and actively participate in Community program design and discussion.