Challenge and analysis of distributed database __ Database

Author: Lu Yi Lei
Author Introduction: Admaster Technology vice President, senior large data technology experts. Focus on high reliability, high availability, high scalability, high performance system services, and focus on off-line, streaming and real-time distributed computing techniques such as hadoop/hbase/storm/spark/flink/elasticsearch.
Editor: Zhonghao (zhonghao@csdn.net)
Copyright NOTICE: This article is "programmer" original article, not allowed to reprint, subscribe to 2016 programmers please click http://dingyue.programmer.com.cn

With the continuous evolution of the demand for large data technology, distributed database is becoming more and more important in the whole ecological circle, and it is expected to become another core of the future development of large data technology, and OLAP (On-line Analytical Processing) is especially important. Basic Theory basic theory of the database acid

Atomic Sex (Atomic). All operations in the entire transaction are either complete or incomplete and cannot be stalled in the middle of the process. An error occurs during the execution of a transaction and is rolled back (Rollback) to the state before the transaction began, as if the transaction had never been performed.

Consistency (consistent). Database integrity constraints are not compromised until the transaction is started and after the transaction ends.

Isolation (Isolated). Quarantine States perform transactions so that they appear to be the only actions that the system performs within a given time. If there are two transactions running at the same time and performing the same function, the isolation of the transaction will ensure that each transaction is considered only in the system by the transaction. This attribute is sometimes referred to as serialization, in order to prevent confusion between transaction operations, the request must be serialized or serializable so that only one request at a time is used for the same data.

Persistence (durable). After the transaction completes, the changes that the transaction made to the database are persisted in the database and are not rolled back.

There are two main ways to implement acid, one is in a log format (Write ahead logging), and almost all database systems (MySQL, Oracle, etc.) are based on journaling. Another is shadow paging, the representative of the database is mainly sqlite,android or iOS app development should be more understanding, but large databases are not used.

Figure 1 Transaction Isolation Overview

cap Theory of distributed database

Consistency (C). The value of all data backups in a distributed system is the same at the same time.

Availability (A). When a subset of nodes in a cluster fails, the cluster as a whole can respond to the client's read and write requests (availability includes not only read but also write).

partition tolerance (P). If some nodes in the cluster are unable to contact, the cluster as a whole can continue to service.

Fig. 2 The three big laws of cap

NoSQL Classification

If you meet these three points, the cost will be very high, so we recommend that according to the specific needs of the business to do a balanced choice. A simple classification of NoSQL will include the following:

Key/value or ' The Big hash table '. Typically there are Amazon S3 (Dynamo), Voldemort, Scalaris, Memcached (in-memory key/value store), Redis, and so on.

Schema-less. Typically there are Cassandra (column-based), CouchDB (document-based), MongoDB (document-based), neo4j (graph-based), HBase ( column-based), Elasticsearch (document-based) and so on. comparative analysis of OLTP and OLAP

At present, the distributed database mainly divides into two kinds of scene--OLTP (on-line transaction processing) and OLAP (On-line Analytical Processing). With the development of large data technology, database selection is more and more, the main differences include: transaction oriented or analysis oriented, data content is current, detailed data or historical, aggregated data, database design is entity Contact Model ER and application-oriented database design, or star type, Snowflake model and theme-oriented database design. The former refers to an OLTP scenario, which refers to an OLAP scenario.

Table 1 Comparison of OLTP and OLAP

Based on the theory of distributed database, whether the optimization of the database or the design, use, there are many problems encountered. For example, now hardware development is very good, special SSD, if its equipment performance is far from achieved, then use SSD database performance how to improve. If only to meet the business's current simple requirements, can be a lot of database transfer engine storage directly into SSD, can quickly solve a big problem. In addition, there is a classic problem, how to ensure the high reliability of the premise of improving database insert and query performance. Just said is a stand-alone mode, multi-machine distributed mode and how to improve the performance of data invocation, there are many challenges. In short, must be based on the needs of the business to choose the most suitable for their own database system. Real case of distributed database HBase

In the hbase design principle, each column family can contain any number of columns, each column can have any number of versions, the column exists only when there is a value, the column itself is sorted.

Focus on the zookeeper model, which uses a very classical model called Leader/follower. For example, when you go to a restaurant, there must be a foreman taking you over, arranging a seat, ordering is not his job, and it's done by his colleagues, which is a very traditional semi synchronous model. and Leader/follower model is the foreman to take you to the past to help you order, he will choose a follower before the leader, through the elective to achieve, this will reduce the scheduling of the thread, which will greatly improve the performance of the database.

Fig. 3 Function implementation in HBase

Elasticsearch (ES)

For the distributed database Elasticsearch also as a distributed database for a reason, if you need a quick query, but a lot of columns, hbase SQL support is not very good, inconvenient to use. and ES for front-end engineers development is very simple, do not need to understand the core of the distributed database can be quickly used, but only need to understand restfulapi, and also very convenient. The ES Bottom is distributed lucene, such as GitHub uses Elasticsearch to search for 20TB of data, including 1.3 billion of files. ES model is relatively clear and simple, on two steps, a step is how to build the data index, the completion of the index is mainly to do the query, how to do the SQL statement query.

Figure 4 Elasticsearch Highlights

The most important thing is to build the index, each record will be indexed according to the requirements, so there are good and bad--if a sudden 10 billion records, the time to build the index is very long, for the business index is intolerable. So if you support offline indexing, then real-time incremental indexing is better, and the ES model is not yet supported. But ES now developed more mature, can be docking interface can support, so it is very convenient.
Comparison of distributed database systems

Figure 5 Elasticsearch Function module

This paper mainly compares Pinot and Druid, and supports the distributed system of multidimensional real-time query.

Table 2 Comparison of Druid and Pinot function implementations

Since the pinot is similar to the ES system architecture, and the Pinot supports a more storage format than Druid, we do a performance test comparison with Pinot and ES, and the test conditions are as follows:

The number of records is divided into 10 billion and 100 billion articles.

The number of servers is 70 units, configured to: 12 cores CPU, memory 96G, hard disk 48T

Test statement: SELECT COUNT (*) from Test where age > Gender > 0 and OS > "+" and SC in ("0001009", "0002036", "00160 "," ... ") or bs>585 and group by Age,gender,os,bs

A total of 12 columns: Dynamic column 3 (multivalued column), General column 9

Table 3 Comparison of Elasticsearch and Pinot Bai retrieval

Table 4 Comparison of Elasticsearch and Pinot Ching retrieval

For Pinot and Es there is a common denominator, they have a multivalued column of attributes, that is, similar properties can be placed in the same column, so that most of the need to find a column of data, which is more beneficial to performance. Real Case Analysis

Business requirements:

More than 10 billion requests per day

Grow more than 5TB per day

1000 dimension calculations per day for hundreds of millions of records

Customers have streaming, real-time, off-line requirements

Figure 6 is the system data flow.

Figure 6 System Data Flow

Data acquisition using WebService, such as Nginx, data collection services with Kafka and Flume, the use of Storm Storm, the main for the following two reasons, the business needs in the millisecond needs; there are strict time series, such as the original input is 1, 2, 3, 4, 5, the output must also be 1, 2, 3, 4, 5. Other use of spark streaming will be better.

Next to the Kafka streaming out of the data corresponding to each different business, and then import the corresponding storage, such as HBase, HDFs, through different streams to solve different business problems, and then based on different storage to do a variety of algorithm analysis Finally, the data of various results are imported into Elasticsearch or MySQL to make the data visualization of the front-end.
By reading the above knowledge I believe you have a certain understanding of the development of distributed databases and the technical characteristics of different systems, limited to the reasons for space, the author to share several es of the use of experience to end:

Using the ES's Alias attribute to realize the horizontal extension of the data.

With Kibana analysis and presentation of data (elk) can meet the needs of many companies more than 80% of the business, elk refers to Elasticsearch, Logstash, Kibana, respectively, they are: Elasticsearch is responsible for log retrieval and analysis Logstash responsible for the collection, processing and storage of the logs; Kibana is responsible for the visualization of the logs and recommends the KIBANA4 version.

Multi-Conditional aggregation query, Boolean query.

Custom participle plug-in (IK), to achieve the exact matching of special characters, now the mainstream search engine in search of keywords when the punctuation is ignored, but in the implementation of some of the monitoring of micro-blog and other social data, if the micro-blog has many symbols, for example, ":" Actually represents a smiley face, The smiley face is very useful for us to judge the negative, so it is not only semantic analysis, but also the analysis of punctuation is very important.