The secrets behind cloud computing yuntable Story

In a previous article in this series, and you mentioned that the world has a lot of nosql products, then why the author on the basis of these products, research and development of new NoSQL database? Because in the development of Yunengine, the author found in the industry also lacks a very concise architecture, and can adapt to a variety of cloud computing scene of the NoSQL database, so at that time I began to carry out yuntable development work. Yuntable's goal is not to do a database like bigtable all-inclusive, but to do a compact version of the distributed Key-value database, the upper layer of cloud computing applications will be based on their own needs to use yuntable or modify, So that yuntable can adapt to the various scenarios of cloud computing, and very easy to use. Yuntable was officially open source in early October and released its 0.8 edition, the official address is http://code.google.com/p/yuntable/. Yuntable is analyzed and described below, including its design, architecture, and how to adapt to different cloud computing environments.

Design of Yuntable

When it comes to the design of a NoSQL database, there are three aspects of data model, distributed architecture and data storage.

In the data model, Yuntable is Key-value, although key-value this data model is relatively simple in structure and traditional relational type, a bit similar to common Hashtable, a key corresponds to a value, but it can provide very fast query speed, Large data storage and high concurrency operation, and is very suitable for the primary key (keys) to query and modify the data, and other operations, although not to support complex operations, but can be through the development of the upper level to make up for this defect.

In the context of distributed architecture, Yuntable chose the single master mode to manage the entire cluster, although this pattern had the potential to be simple in both implementation and semantics, and to avoid a single point of failure in master, Yuntable will introduce the shadow-master mechanism in future releases.

In the area of data storage, Yuntable chose the sstable file format. In simple terms, sstable is a file format for storing sorted key-value pairs, and is immutable (immutable), that is, after it is written, only its updates are appended to it, and cannot be modified directly. This is to allow the system to perform sequential access, rather than random access, that disk is good at. In the internal format, the Sstable file consists of both the index and the data block. In actual operation, the system often loads the index into memory to ensure the efficiency of the query.

Yuntable Architecture

In terms of architecture, it can be divided into three modules, region, master, and client, and these three modules are independent and responsible for their respective business logic.

▲ Figure 1. Yuntable Architecture

First of all, introduce the master node, the master node in the function is relatively "light" tune, mainly responsible for maintaining the corresponding relationship between table and region nodes, the actual data query and input are through the region node and the client side of the interaction between the completion, Regardless of the master node, this can effectively reduce the burden on the master node and enable it to support more than one cluster on hundreds of servers. For example, when a client side needs to process a table, it simply asks Master for the address of the region node associated with this table the first time it is processed, and when it is processed again to this table, The client side does not need to communicate with the master node, but directly interacts with the associated region node.

Second, talking about the region node, which is responsible for processing requests from the client side and storing and managing large amounts of data, region nodes are very similar to the tablet servers mentioned in BigTable papers. Each region server manages multiple tablets, each tablet corresponds to a table, and is responsible for storing data that belongs to this table. In addition to managing multiple tablets, the region server also comes with Wal logs, called "Write-ahead Log", which are used primarily for staging those latest data update requests to avoid data loss when the Memstore in the tablet is accidentally closed. When Memstore completes writing to the data, Wal also empties the corresponding data. The tablet that is used to store the data consists of two main parts: one is Memstore: It is a data file that is cached in memory, which mainly stores the most recently added data, and the data cached on Memstore will be flushed when the Memstore store data is close to the limit (Flush) To Yfile, the second is yfile, it is mainly used to store data persisted file, it is based on the sstable format mentioned above, Yfile will only be created when the memstore is triggered flushing, usually in order to read, this can effectively use the hard drive sequential read performance characteristics, The location of the file is in the directory of the tablet that it belongs to.

Now the client side is mainly called "yuncli" command line, mainly for users to input data-related commands, and with the back-end of the master node and region node interaction, but with the development of time, in the form, The client side may be a JDBC-like driver.

How to adapt to different cloud computing environments

The main common types of cloud computing are two scenarios: requires low latency and high concurrent read and write capabilities, large amounts of data, but not Shanghai, estimated at up to TB levels, most of the Web applications now using RDBMS fall into this category, somewhat similar to traditional OLTP (online transaction processing); Storage and operation of massive data , such as PB-level, this example has a traditional data warehouse, Google massive web pages and image storage, a bit similar to the traditional OLAP (online analytical Processing).

So how does yuntable adapt to both environments? First, stick to the classic and universal design of Key-value, Single-master and sstable. Next, in the data storage aspect, joins the hotness this mechanism, mainly is by sets the hotness value to decide before to complete the query to read to the memory data block's life time, assumes if is the low delay condition, then sets the hotness value to be long, if is the massive data , the opposite.

Finally, yuntable as a new generation of PAAs platform Yunengine of the back-end database has been put into operation, and is about to release its version 0.9, in this release, Yuntable's single point of performance and stability will embark on a higher level. Also, the next article will continue to pay attention to NoSQL.

Author Introduction

Wu Zhuhua, who previously participated in the development of several cloud computing products at the IBM China Academy, now focuses on yuntable (http://code.google.com/p/yuntable/) and Yunengine (http:// yunengine.com/) research and development, and is about to publish "Analysis of Cloud Computing" a book, please look forward to.