This article was reproduced from: http://coolshell.cn/articles/7270.html
================================================
Full-text translation from the wall of the article "NoSQL Data Modeling techniques", translation is not good, please forgive me. After reading this article, you may feel a little bit about NOSQL data structures. My feeling is that the relational database wants to do a good job of consistency, integrity, indexing, crud, NoSQL doing only one thing, but sacrificing a lot of other things. Overall, I think NoSQL is a better place to be a cache. Here is the text--
NoSQL databases are often used as a place for many non-functional places, such as extensibility, performance, and consistency. These NoSQL features are being extensively researched in theory and practice, and the focus of research is on non-functional things related to performance distribution, and we all know that the CAP theory is well used in NoSQL systems (Chenhao Note: cap is, consistency (consistency), Availability (availability), partition tolerance (Partition tolerance), in distributed systems, these three features can only be implemented at most two simultaneously, and NoSQL generally abandons consistency. On the other hand, NoSQL data modeling techniques are not well researched by the world because of the lack of basic theories like relational databases. This article compares the NoSQL family with data modeling and discusses several common data modeling techniques.
To start discussing data modeling techniques, we have to look at the growing trend of NOSQL data models more or less, so that we can understand some of their inner connections. Is the evolution of the NoSQL family, and we can see this evolution: The Key-value era, the BigTable era, the document age, the full-text search era, and the graph Database era: (Chenhao Note: Note that the SQL said in the figure of the sentence, The way NoSQL goes on is SQL, haha. ）
NoSQL Data Models
First, we need to note that the SQL and relational data models have been around for a long time, and this nature of user-oriented means:
SQL, on the other hand, allows software applications to control data aggregation and data integrity and effectiveness in many cases without concern for the database. And if we go beyond data consistency, the integrity of these things can be a heavy help for performance and distributed storage. Because of this, we have the evolution of the data model:
The remainder of this article will introduce you to the technical implementation and related patterns of data modeling. However, before introducing these techniques, let's start with a preface:
Here's a list of NoSQL categories, and the products I used to do when I wrote this article:
This section focuses on the basic principles of the NoSQL data model.
(1) Anti-normalization denormalizationAnti-normalization denormalization can be thought of as copying the same data into different documents or tables, which simplifies and optimizes the query, or just fits the user's particular data model. Most of the techniques described in this article are more or less guided by this technique.
In general, anti-normalization needs to weigh these things:
Applicability : Key-value Store Key-value pairs database, document databases documentation database, bigtable-style database.
(2) Polymerization aggregatesAll types of NoSQL databases provide a flexible schema (data structure, restrictions on data formats):
The flexible schema allows you to store a set of associated business entities in a nested internal data way (Chenhao Note: A data encapsulation format similar to JSON). This can bring us two benefits.
Both of these benefits are signalled. The picture depicts the commodity model in e-commerce (Chenhao Note: I remember the challenge of the product Classification database design in the "challenges everywhere" article)
For relational databases, it is not easy to design such a data model, and the design is absolutely far from elegant. And the flexible schema of our nosql allows you to use an aggregation Aggregate (product) to build all the different kinds of goods and their different attributes:
Entity Aggregation
We can compare the differences between relational databases and NoSQL. but we can see that non-normalized data storage has a big impact on performance and consistency on data updates, which is where we need to focus and sacrifice .
Applicability : Key-value Store Key-value pairs database, document databases documentation database, bigtable-style database.
(3) Application layer coupling application Side JoinsTable joins are largely not supported by NoSQL. As we said earlier, NoSQL is "problem-oriented" rather than "face-to-answer", and not supporting table junctions is the consequence of "problem-oriented". The junction of a table is constructed at design time, not at execution time. So, table joins at run time is very expensive (Chenhao Note: The SQL table junction knows what the Cartesian product is, big can be seen in the previous cool shell "diagram database table joins"), but after using denormalization and aggregates technology, We basically do not have to do table joins, such as: you use nested data entities. Of course, if you need to connect data, you need to do it at the application level. Here are a few of the main use case:
Applicability : Key-value Store Key-value pairs database, document Databases documentation database, bigtable-style database, graph Databases graph database.
Universal modeling Technology General Modeling techniquesIn this book, we will discuss various common data modeling techniques in NoSQL.
(4) Atom polymerization Atomic AggregatesMany NoSQL databases (not all of them) are short-board in transaction processing. In some cases, they can be implemented through distributed lock technology or application layer-managed MVCC technology (Chenhao Note: See the "Multi-version concurrency control (MVCC) application in Distributed Systems") however, In general, only the polymerization aggregates technique can be used to guarantee some acid principles.
This is why our relational database requires a strong transaction mechanism-because the data of the relational database is normalized and stored in a different place. So, aggregates aggregation allows us to save a business entity as a document, save it as a single line, and save it as a key-value so that it can be atomically updated:
Atomic Aggregates
Of course, atomic aggregation Atomic aggregates this data model does not implement transactional processing in a full sense, but if you support atomicity, locks, or test-and-set directives, then Atomic aggregates can be applied.
Applicability : Key-value Store Key value pairs database, document databases documents database, BigTable style database.
(5) Enumerable key Enumerable keysPerhaps the biggest benefit for Key-value is that business entities can be easily hashed to partition on multiple servers. The sorted key will complicate things, but sometimes an app can get a lot of benefits from sorting keys, even if the database itself doesn't provide this functionality. Let's consider the data model for the email message:
Applicability : Key-value Store Key-value pair database .
(6) Reduced dimension dimensionality ReductionDimensionality Reduction is a technique that allows a multidimensional data map to be mapped into a key-value or other non-multiple data model.
The traditional geographic information system uses some such as "four-tree Quadtree" or "r-tree" to do geo-indexing. The content of these data structures needs to be updated in the appropriate location, and if the amount of data is large, the cost of operation will be high. Another way is that we can traverse a two-dimensional data structure and flatten it into a list. A well-known example is Geohash (geo-hash). A geohash uses the "zigzag" route to scan a 2-dimensional space, and the movement in the traverse can be simply represented by 0 and one, and then 0/1 strings are generated during the move. Show this algorithm: (Chenhao Note: First to divide the map into four parts, longitude is the first, latitude is the second, so the longitude on the left is 0, the right is 1, the same latitude, the above is 1, the following 0, so that the latitude can be combined into 01,11,00,10 this four values, which identifies four areas, We can do this constantly recursively four points per region, then we can get a string of 1 and 0, and then use 0-9,b-z to remove (remove a, I, L, O) the 32 letters for BASE32 encoding to get a 8-length encoding, which is Geohash algorithm)
Geohash Index
The most powerful feature of Geohash is the ability to know the distance between two regions using a simple bit operation, as shown in the figure (Chenhao: The two of the proximity box, which is much like an IP address). Geohash has transformed a two-dimensional coordinate into a one-dimensional data model, which is the dimensionality reduction technique. BigTable's dimensionality reduction technology See [6.1] later in the article. More about Geohash and other technologies can be found in [6.2] and [6.3].
Applicability : Key-value Store Key value pairs database, document databases documents database, BigTable style database.
(7) Index TableIndex table is a straightforward technique that allows you to get the benefits of indexing in a database that does not support indexing. BigTable is the most important database of its kind. This requires that we maintain a special table with corresponding access patterns. For example, we have a main table with a user account, which can be accessed by the UserID. A query needs to find out all the users in a city, so we can add a table, this table with the city key, all and this city is related to the UserID is its value, as follows:
Index Table Example
It is visible that the needs of the City Index table and the user table for the main table are consistent, so each update of the primary table may need to be updated on the index table, or it is a batch update. Either way, this can damage some performance, because consistency is required.
The index table can be considered the equivalent of a view in a relational database.
Applicability : BigTable database.
(8) Key combinations index Composite key indexThe Composite key combination is a very common technique, which can be very beneficial when our database supports key ordering. Composite key combination is a second sort field that allows you to build a multidimensional index, much like the dimensionality Reduction technology we said before. For example, we need to access user statistics. If we need to count the distribution of the users according to different regions, we can design the key into such a format (State:City:UserID), so that we can go through the state to the city to traverse the user by group, In particular, our NoSQL database supports query-by-zone on key (e.g., BigTable class system):
1
2
SELECTValues WHEREstate="CA:*"
SELECTValues WHEREcity="CA:San Francisco*"
Composite Key Index
Applicability : BigTable database.
(9) key combination aggregation Aggregation with Composite keysThe Composite keys combination technique can be used not only for indexing, but also for distinguishing between unused types of data to support data grouping. Consider an example where we have a huge array of logs that record the source of access for users on the Internet. We need to calculate the number of independent visitors coming from a site, and in a relational database, we might need the following SQL query statements:
1
SELECTcount(distinct(user_id)) FROM clicks GROUP BYsite
We can create the following data models in NoSQL:
Counting Unique Users using Composite Keys
In this way, we can sort the data by UserID, and we can easily process the same user's data (one user does not generate too many event), and remove the duplicate sites (using hash table or whatever). Another alternative technique is that we can create a data entity for each user and then append its site source to the data entity, which, of course, will have a loss in performance compared to the data update.
Applicability : Ordered key-value Store sort key value to database, BigTable style database.
This technology is more data-processing technology, rather than modeling technology. Still, this technique will affect the data model. The main idea of this technique is to use a lead to find data that satisfies a condition, but aggregating the data requires a full-text search. Or let's say an example. Or with the example above, we have a lot of logs, including Internet users and their sources of access. Let's assume that each record has a userid, as well as the type of user (men, women, Bloggers, etc.), the city where the user is located, and the site visited. What we want to do is to find individual users who meet certain criteria (access source, city, etc.) for each user category.
Obviously, we need to search for those who meet the criteria, and if we use reverse search, this will make it easy for us to do this, such as: {Category--[user IDs]} or {Site-and [user IDs]}. With such an index, we can take the intersection or the set of two or more userid (which is easy to dry and can be done quickly if the userid is in order). However, it can be a bit cumbersome for us to generate reports by user type, as we may use statements like this
1
SELECTcount(distinct(user_id)) ... GROUP BYcategory
But this kind of SQL is inefficient, because there are too many category data. To deal with this problem, we can create a direct index {UserID-and [Categories]} and then we use it to generate the report:
Counting Unique Users using inverse and Direct Indexes
Finally, we need to understand that random queries on each userid are inefficient. We can solve this problem by batch query processing. This means that for some user sets, we can preprocess (different query conditions).
Applicability : Key-value Store Key-value pairs database, document databases documentation database, bigtable-style database.
Hierarchical model Hierarchy Modeling techniques (11) tree-shaped aggregation trees AggregationTree-shaped or arbitrary graphs (which need to be normalized) can be directly placed into a record or document.
Tree Aggregation
Applicability : Key-value Key-value pair database, document Databases documentation database
(12) adjacency list adjacency ListsAdjacency Lists adjacency list is a graph – each node is a separate record that contains all the parent nodes or sub-nodes. In this way, we can search by a given parent or child node. Of course, we need to traverse the graph through hop queries. This technique is inefficient in the breadth and depth of queries, and the subtree that gets a node.
Applicability : Key-value Key-value pair database, document Databases documentation database
Materialized Paths can help avoid recursive traversal (e.g., tree structure). This technique can also be thought of as a variant of anti-normalization. The idea is to add the identity attribute of the parent node or the child node to each node, so that you know all descendant nodes and ancestor nodes without having to traverse it:
Materialized Paths for EShop Category Hierarchy
This technique is very helpful for full-text search engines because it allows you to turn a hierarchy into a document. In the above illustration we can see that all the products or subcategories under the men's Shoes can be processed by a very short query statement-only given a category name.
Materialized Paths can store a collection of IDs, or a bunch of string IDs to spell out. The latter allows you to search for a specific branch path through a regular expression. This technique is shown (the path of the branch includes the end point itself):
Query materialized Paths using REGEXP
Applicability : Key-value key value to database, document Databases documentation data, search Engines search engine
(14) Nested set Nested setsNested sets nesting set is the standard technique of tree structure. It is widely used in relational databases and is fully applicable to Key-value key-value pairs database and document Databases documents database. The idea of this technique is to store the leaf nodes in an array and map each non-leaf node to a leaf node set by using the start and end of the index, as shown here:
Modeling of ECommerce Catalog using Nested sets
Such a data structure is very efficient for the immutable of the database, because its point memory space is small, and can quickly find all the leaf nodes without the need for tree traversal. However, there is a high performance cost to insert and update because the new leaf nodes need to be updated on a large scale.
Applicability : Key-value Stores Key value database, document Databases documentation database
(15) Nested Document flattening: Limited field name Nested documents flattening:numbered Field NamesSearch engines basically work with flat documents, such as: Each document is a tabular table of fields and values. This data model is used to map business entities to a text document, which can become challenging if your business entity has a complex internal structure. A typical challenge is to map a hierarchy of documents. For example, a document is nested within another document. Let's take a look at the following example:
Nested Documents problem
Each of these business entity codes is a resume. It includes a name and a list of skills. I mapped this hierarchical document into a text document, one way to create the Skill and level fields. This model can search for a person by technology or rank, and the combination query that is labeled will fail. (Chenhao Note: Because it is not clear whether the excellent is math or poetry)
[4.6] In the reference gives a solution. Each field is labeled with the number skill_i and level_i, so that each pair can be searched separately (using or to traverse the lookup for all possible fields):
Nested Document Modeling using numbered Field Names
There is no extensibility in this way, and for some complex problems only the complexity of the code and the maintenance work are made larger.
Applicability : Search Engines Full Text Search
(16) Nested document flattening: Proximity query Nested documents flattening:proximity QueriesThis technique is used to solve flat-level documents in Appendix [4.6]. It uses neighboring queries to limit the range of words that can be queried. , all skills and levels are placed in a field called Skillandlevel, and the "excellent" and "poetry" that appear in the query must be followed by another:
Nested Document Modeling using Proximity Queries
Appendix [4.3] Describes a successful case in which this technique was used in SOLR.
Applicability : Search Engines Full Text Search
(17) Graph Structure Batch graph processingThe graph Databases graph database, such as NEO4J, is an excellent graph database, especially when you use a node to explore neighbor nodes, or to explore relationships before two or fewer nodes. However, processing large amounts of graph data is inefficient because the performance and extensibility of the graph database is not its purpose. Distributed Graph data processing can be handled by the MapReduce and Message passing pattern. For example, in my previous article. This method can make Key-value stores, Document databases, and Bigtable-style databases suitable for processing large graphs.
Applicability: Key-value Stores, Document Databases, Bigtable-style Databases
ReferenceFinally, I provide a list of useful links related to NoSQL data modeling:
(End of full text)
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
1 on 1 presale consultation
24/7 Technical Support 6 Free Tickets per Quarter Faster Response
Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.
A comprehensive suite of global cloud computing services to power your business
Payment Methods We Support
