Thinking in BigDate (7) Big Data technology core NoSql (1)

Thinking in BigDate (iii) mechanism and trend of big data operation. This article talks about the challenges of structured data. With the increasing volume of data,NoSqlIt came into being. In this article, we will unveil the secrets of NoSql and open the door to unstructured data (no more discussion of the characteristics of relational databases ).

The original Article is summarized in NoSql Distilled by PramodJ. Sadalage, Martin FowlerLove flying translate NoSql Essence

Learning is equivalent to Learning white. The focus is thinking!

NoSql (1): outline:

Data collection:ETL

Data storage:Relational databases, NoSql (Not Only SQL), SQL, etc.

Data management:(Infrastructure Support) cloud storage and Distributed File Systems

Data analysis and mining:(Computing Result Display) Data Visualization

Origin:

Traditional relational databases may have been used in the business world, but as the amount of data to be processed increases, the number of cluster data increases. Traditional databases cannot meet the requirements of different data formats. As a result, we began to explore and look forward to a big data processing technology that is easy to program and highly efficient in the cluster environment. In this case, NoSql came into being.

Why are bottlenecks in traditional relational databases? Why is the increase in data accompanied by the increase in non-structural data? What is non-structural data? What is NoSql? Which databases support NoSql data? What is the difference between NoSql databases and relational databases? What are the challenges of NoSql? Why does it attract attention? Who should pay more attention to NoSql? And so on.

Maybe this is the birth of a new thing, which will inevitably lead to the consistent pursuit of good ideas, asking various questions you can imagine, and trying every means to push it forward. It is also possible that in this circle, there will always be a group of tireless people constantly exploring and constantly seeking for solutions to change the world, which is also a great pleasure for these people.

The reason for the rapid development of NoSql: as more and more data needs to be processed, the expansion methods of large systems have beenVertical Scaling (Scale up)ToHorizontal scaling(Scale out). This also applies to a number of NoSql database data models have an important feature, that is:Data closely related to the content can be organized into a rich structure and displayed and stored as a unit (Unit) to access. This structure is now called aggregate ).

Comparison Between Relational databases and non-relational databases: relational databases

Features:

1. persistent data: It can persistently store a large amount of data. In this case, "primary storage = memory" and "backup storage = Hard Disk" exist ". The hard disk can be used for persistent data.

2. Concurrency: it is not extremely difficult for operations to obtain data in the database. The "transaction" (with "transaction errors") processing mechanism can ensure that data is not damaged and solve the concurrency problem.

3. Integration: Shared data integration: stores data of multiple applications in the same database. It is conducive to team development. At the same time, it can cope with the concurrency mechanism of multiple applications.

4. Standard Model: This is reflected in database standardization. It helps developers and database experts to learn basic Relational Models and apply them to different projects. Although there are differences between different databases, the core mechanism is the same, and the SQL and "transaction" operations are almost the same.

Disadvantages:

1,Impedance detuning:To the developer's disappointmentRelational Database ServiceInLINK modelAndData Structure in memoryBetweenDifferenceThis difference is often calledImpedance detuning".LINK modelOrganize data"Table" and "row ),(You can understand that "tables" and "rows" are too standard.),More accurately, it should beRelation and tuole). In the relational model,A set of key-value pairs, AndA link is a set of tuples.(If you know NoSql a little bit (NoSql is defined as a non-relational database for the moment) based on key-value, it is not hard to understand it above)

2,Cluster:Cannot run in the Cluster.As the number of Internet companies rapidly increases. How to solve storage problems? This introducesVertical and horizontal scaling. Vertical: more powerful computers are needed. Horizontal: a cluster composed of multiple small computers. Without a doubt, it can reduce costs and improve performance. This causes the following problems:Relational databases are not used for cluster design., They need a waySupports cluster file systems,File SystemYesWrite Data to the available disk subsystem at any time.

Relational Database ServiceYou can also divide the data into several sets.AndAllocated to independent serversRun, so you can effectively shard the database. Although the load can be distributed to multiple serversThe program must control all parts., ItYou need to know the server on which each piece of data is stored in the database.. Data Integrity and other properties must be considered. In this case, we can only say that this is a "extraordinary way", and it is imperative.

Non-Relational Database Service (NoSql ):

Features:

1. NoSql databases do not use or do not use SQL. It cannot be said that it is an advantage or a disadvantage. The problem is that most people are used to SQL and suddenly don't use it. Isn't it forcing people to change it? You know, they're very lazy. This produces someSupport for SQL plug-insThe birth of, don't underestimate, this is enough to surprise you (Introduction later).

2. Open Source (key)

3. abandoned the relational model.

4. can run in the Cluster

5,"No mode" DataYou can add fields without modifying the structure definition.

6. Not Only SQL. "It's not just SQL". In this way, NoSql is considered not only a technology, but alsoChange.

Disadvantages:

1. Since NoSql technology is not yet mature (recently, such as Cassandra, MongoDB, Neo4J, and Riak), most enterprise-level application developers should also focus on existing relational databases, however, in highly prospective projects, you can choose to try first.

2. There is no standardized definition. The models of each NoSql solution (such as "key value", "document", "column family", and "Graph") are different,Causes compatibility differences between NoSql operation statements of different versions.

Two concepts:

1,Data Model:It is the model used for cognitive and operational data. The data model describes how to deal with data in the database. We are more concerned with data models.

2,Storage Model:Describes the mechanisms for storing and operating data in the database. Generally, this parameter is not involved.

Aggregate Data Model: Aggregation

The relational model divides the information to be stored into a metagroup (all users familiar with database operations know the row:Relational databases operate on rows. Meaning:Read/write operations by row)Tuples are restricted data structures.: It can only contain a series of values, soAnother unary group cannot be nested in the tuples., AlsoCannot contain a list composed of values or tuples. All operations are requiredTarget with tuplesAndThe returned value must also be a tuple..

Aggregation-orientedThe usage is different.Units used to operate dataIts structure is much more complex than that of the tuples. If the list or nested other record structures can be stored in this complex structure, the above problem is solved, "key-value Database", "Document Database", and "column Family database" all use more complex records. Based on this complex record for short, "aggregation ".

To better understand what aggregation is, we provide an example:

Assume that the user records of an e-commerce website must store information such as user information, product catalog, order, shipping address, bill address, and payment method. In this application scenario, we can use both Relational Data Modeling and NoSql data modeling to compare the advantages and disadvantages of the two.

Relational Modeling(Simple example)

1. Ensure that no duplicate data exists between tables.

2. Maintain "integrity" between tables"

Figure Data Model for relational databases

Figure sample data

Modeling using an aggregate model:

Figure aggregation data model black diamond represents the relationship between each data in the aggregation structure

Sample Data is represented in JSON format because it is a common data format in the NoSql field.

// in customers{"id":1,"name':"Martin","billingAddress":[{"city":"Chicago"}]} // in orders{"id":99,"customerId":1,"orderItems":[    {    "productId":27,    "price":32.45,    "productName":"NoSql Distilled"    }    ],    "shippingAddress":[{"city":"Chicago"}]    "orderPayment":[    "ccinfo":"1000-1000-1000"    "txnId":"abelif879rft",    "billingAdress":{"city":"Chicago"}    ],}

This model has two major aggregates: Customer and Order ). The customer data includes a bill address list. The order data includes the order item list, shipping address, and payment information ), the payment information includes the billing address corresponding to it.

The same logical address (the Bill address and the shipping address share the "Chicago") inThe sample data appears three times.But here we do not need ID to refer to, but directly copy this address string. If the shipping address and Bill address do not change, it is appropriate to do so. In relational databases, this means that the row of ID = 77 in the Address Table remains unchanged. To change an address, create a new row in the table.After using the aggregation model, we can copy the entire address structure to all the aggregation models. (?)

Considering that the above model has the same trade-offs with relational databases, we can also use unconventional models to directly write product names to order items. This method is common in the aggregation model. We hope to minimize the number of aggregation requests during data interaction. In fact, I can also use another method to place all orders placed by the customer into the customer aggregation.

// in customers{"id":1,"name":"Martin","billingAddress":[{"city":"Chicago"}],“orders”:[{"id":99,"customerId":1,"orderItems":[    {    "productId":27,    "price":32.45,   "productName":"NoSql Distilled"    }    ],   "shippingAddress":[{"city":"Chicago"}]   "orderPayment":[   "ccinfo":"1000-1000-1000"   "txnId":"abelif879rft",   "billingAdress":{"city":"Chicago"}    ],}]}}

Here we must talk about how to divide the aggregation boundary (that is, the part of the red text above). In fact, there is no standard answer to the boundary division During modeling. If you want to access all the customer's orders at one time, you should put them in a large aggregation. Otherwise, if you only want to process one order at a time, the "customer" and "order" should be separated.This depends on the situation, and sometimes this boundary is difficult to distinguish. Sometimes this aggregation will lead to more problems, so aggregation is not suitable.

Impact of aggregation-oriented:

AlthoughLink ingAlsoHandling the relationship between various data elements. The preceding example shows that an order consists of an order item, delivery address, and payment information. In the relational model, you can use"Foreign key"To indicateRelationship between tablesBut this cannot be doneDetermines whether a link can represent aggregation.. Therefore,Databases cannot use aggregate structures to store distributed data.

The deciding factor for selecting an aggregation-oriented model is that it is very suitable for running in clusters.This is the key to NoSql. When running on the clusterThe number of nodes required for data collection is minimized..If the database explicitly contains the aggregation structure and knows which data needs to be aggregated for operation, the data can be placed in a node.(It is very important that the database is distributed across different nodes in the preliminary design. For example, if you deploy MongoDB in a cluster to collect logs from different machines, because the log files collected from different machines are different, you need to consider the problems mentioned above, which data can be collected on a node is more conducive to aggregation,Practical application is critical)

Relational Database SupportTransaction Processing, That is,"ACID transactions"Is the key point in relational databases. So is it suitable for NoSql transaction processing? Generally, aggregated-oriented databases do not support ACID transactions that span multiple aggregates. Instead, an atomic operation can be performed on only one aggregate structure at a time, that is. If we want to perform multiple aggregation operations in an atomic way, we must write the application code by ourselves (Isn't that nonsense?).

Key Value and document data model

Key values and document databases are aggregated-oriented.. These databases are mainly built through aggregation, and both types of databases contain a large number of aggregation, each of which contains the key or ID used to obtain data.

Differences between the two data models:

Key Value database:Aggregation is not transparent (the data structure can be used by external programs without knowing its internal implementation details ). Basically allSearch for aggregated content by keyTo query all the data. You cannot query or retrieve only some of the data. The advantage is that any data can be stored in aggregation, and the database may limit the total size of aggregation.

Document Database:You can see its structure. The keyword used to submit a query is usually based on the document content. You can query and obtain some of the data. It may be obtained by a key, or not by a key. Advantage: although the content in the database is restricted and its own structure and data type are defined, this advantage allows for more flexible access to data.

Column family Storage Model

Early NoSql databases were Google's BigTable. The emergence of the BigTable model affects NoSql databases such as HBase and Cassandra. At first, mostDatabases store data in behavior units, Especially in the need to improveWrite Performance (many write operations). However, in some casesFew write operations are performed, But often needRead multiple columns in a row at a time. In this caseA group of columns of all rows as basic data storage units, The effect will be better. So the name of "column store database" came.

Both BigTable and its successors follow the"Stored in a group of columns (that is, columnfamily). Later, most NoSql databases can be called"Column Family database" or "column type Database" = "graph database".

The best way to understand the column family model is to treat itTwo-level aggregation structure(Two-level aggregate structure ).Like "key-value storage", the first key usually represents a row identifier,UsedObtainWhat you wantAggregation.The difference between "key-value storage" is that its "Row aggregation" is a ing, which contains more detailed values.

The column Family database organizes columns into column families. Each column must be part of a columnfamily, and the data access unit must also be a column. The premise of this design is,Data in a column family often needs to be accessed together..Divides aggregation into columnfamily, Let the database regard itA data unit in a row aggregation..

Figure columnfamily structure indicates customer information

1. row-oriented (row-oriented): each row is an aggregation (for example, a customer with ID-123456 is an aggregation ), this aggregation contains some columns of useful data blocks (customer information, order records.

2. column-oriented (column-oriented): Each column Family defines a record type (for example, customer information), where each row represents a record. (The "row" here refers to the row in the above figure, which actually corresponds to the column family in the database)

Column orientation reflects the importance of columns in the column Family database. Because the database understands thisCommon Data grouping MethodsTo use this information () for storage and access ().You can add columns as needed in the column family. When creating a project list, you can represent each project as a separate column.

Summary:

Above, we roughly introduce three different styles of data models oriented to aggregation.In common: the cluster runs, and aggregation is the central link, because the database must ensure that the aggregated data is stored on the same node.Aggregation is the smallest data unit for "Update" operations. For transaction control, aggregation is used as the operation unit and the size is exactly the same.

·Aggregation is the data set of interaction units.ACID operations in databases are bounded by aggregation.

·Key-Value Data Reading, document databases, and column family databases are all oriented towards aggregate databases.

·Aggregation makes data storage management more effective for databases in clusters.

· If data interaction is mostly performed in the same aggregation, you can use an aggregation-oriented database. If data in different formats is required for interaction, it is best to use a non-aggregation database.

We will continue to explore the NoSql mechanism and implementation in the future.

CopyrightBUAA