NoSQL Selection and HBase case

Next, Wang Shupeng, associate researcher of Institute of Information Engineering, Chinese Academy of Sciences, shared the development and use of new NoSQL large data management systems (BDMS). Wang Shupeng said most of the projects he contacted were non-Internet applications, such as security and transportation. These industries are now faced with big data, but many popular NoSQL databases do not apply to them, so they independently developed a NoSQL database management system.

Design Objectives

The system is highly scalable: it can be linear by adding nodes

Support for unified storage management of complex data types: structured, semi-structured and unstructured data, text data, multimedia data, and unified organization management and processing for multiple types of business data

Support for diverse types of access, Access interface standardization: retrieval, statistical analysis, association processing and in-depth mining, the need for a variety of business data related to comprehensive analysis, the provision of standard DDL, DML operation syntax, support JDBC, ODBC and other operational interfaces, data retrieval, statistics, analysis and processing of real-time requirements are high ; retrieval requires a second level response; cross-domain Retrieval access

The diagram above is the framework of the whole system, in which the structure of the database management platform is as follows:

This can be done across data management through the management engine. External can provide the corresponding DDL interface, DML interface and development interface.

Main features of the system

Share-nothing distributed storage and computing architecture

Organization and management of heterogeneous multi-source data: Unified storage management of structured data, unstructured text and unstructured multimedia

Unified SQL query supporting heterogeneous data: support for the retrieval and analysis of structured and unstructured text, which can be implemented through SQL

Rich data access and processing patterns

Efficient retrieval mechanism

Heterogeneous multiple replica storage and recovery mechanisms

Cross-domain data management and retrieval support for Cross-domain deployments, where multiple data centers can be built in multiple physical locations, enabling data to be moved between data centers, and enabling global retrieval and access to data located in different geographies

Application Scenarios

Management of massive structured records

Handle large amounts of small document management and processing

An intelligent search and mining system for heterogeneous data

Success Stories

Wang Shupeng introduced the system has a successful application case, is a national ministry of a large data management project. The main requirements of this system are:

A large number of information records, generated about 4 billion per day (about 4TB);

Data keep backup copy, record data for six months;

Data can be accurate, fuzzy query and statistics, the results of second-level response;

Structured and unstructured data can be imported in batches;

The final implementation effect is:

A distributed storage architecture (3 meta data nodes + 115 storage nodes) is adopted.

The data scale is over 500 billion, the query response time is second level;

Data is kept 2 copies to ensure data security;

The usable capacity of the system is about 2PB.