MongoDBSharding Mechanism Analysis

MongoDB is a popular non-relational database. As a document-based database, in addition to the flexible data structure with or without schema, MongoDB supports complex and rich query functions, and also comes with a powerful sharding function. To talk about MongoDB sharding, let's first talk about sharding. Sharding refers

MongoDB is a popular non-relational database. As a document-based database, in addition to the flexible data structure with or without schema, MongoDB supports complex and rich query functions, and also comes with a powerful sharding function. To talk about MongoDB sharding, let's first talk about sharding. Sharding refers

MongoDB is a popular non-relational database. As a document-based database, in addition to the flexible data structure with or without schema, MongoDB supports complex and rich query functions, and also comes with a powerful sharding function.

To talk about MongoDB sharding, let's first talk about sharding. Sharding refers to splitting data horizontally to different physical nodes. There are two important points here: horizontal splitting and physical nodes. Generally, there are two database/table sharding types. One is horizontal division, such as modulo by user id and division of user data by remainder, such as blog articles. The other is vertical division, such as placing user information on a node, put the article on another node, or even put the basic information of the article title on one node and the text on another node. Sharding refers to the previous one. Physical nodes are mainly distinguished from Table Partitions provided by mysql. Although Table partitions divide data, these partitions are still on the same physical node.

So why sharding? What problems does sharding solve and what benefits does it bring? Many people have seen their websites and applications grow from small to large, with more and more users, more and more traffic, and more data. This is of course a good thing. However, databases that were previously protected by a server cannot be used now. Optimization can also be done at the beginning, and a cache can be added. However, in any case, a server cannot afford it. The data volume will soon exceed the server's hard disk capacity. At this time, we had to split and perform sharding. Here, sharding can use more hardware resources to solve the performance limit of a single machine. In addition, after the data is horizontally split, the volume of each index is reduced. Because the index of a database is a B-tree structure, the index depth decreases after the index volume is reduced, and the index search speed increases accordingly. For some time-consuming statistical queries, the calculation workload can also be apportioned to multiple machines for simultaneous computation, and the speed can be improved through distribution.

Although sharding has many advantages, traditional databases will encounter a lot of troubles in sharding. The first is expansion and initialization. For example, I used to model 5 by user id and divide it into five nodes. As data grows, these five nodes are not enough and need to be increased. At this time, if the number is changed to 10, at least half of the data needs to be moved. If it is not an integer multiple, such as extending to seven nodes, most of the data will be moved. This is a terrible thing for large-scale databases that have already done sharding. In addition, services are usually unavailable during data migration, which may cause service interruption for a long time. The same is true for databases that have not been sharding. If you use virtual nodes, such as dividing data into 1000 virtual nodes, and then finding the corresponding physical nodes through ing, this can be improved. But it still cannot avoid service interruption during the migration process. Another headache is data routing. After the data is split, the application needs to locate the node where the data is located. You also need to combine the query results involving multiple nodes. If the database middleware is not used, it takes a lot of effort to implement it. Even if the middleware is used, it is difficult to achieve transparency. Due to the complexity of relational database functions, many functions cannot be used normally in sharding. Such as join and transaction. Therefore, it will cause a lot of modification and testing work on the application layer.

Sharding has a lot of troubles. What about MongoDB sharding?

MongoDB sharding features automation. It is embodied in dynamic resizing, automatic data balancing, and transparent interfaces. It can be smoothly upgraded from a common replica set or a single instance. It can dynamically add and delete nodes and respond to rapid growth of data. The data volume can be automatically balanced between nodes to avoid Concentrated Loads on a few nodes. During this period, database read/write access is not affected. The client can use the same driver. Most of the functions are available and no code needs to be changed.

What is the implementation mechanism of MongoDB sharding with such powerful functions? The structure of MongoDB sharding.

We can see that MongoDB sharding is mainly divided into three parts. Shard node, config node, and config node. The client directly accesses the green mongos node in the figure. The config node and shard node on the backend cannot be directly accessed by the client. Mongos is mainly used for data routing. Locate the data location from the metadata and merge the query results. In addition, the mongos node is responsible for data migration and automatic data balancing, and serves as the management node of the sharding cluster. Its external interface is the same as the ordinary consumer D. Therefore, you can use the standard mongodb client and driver for access. Mongos nodes are stateless and do not store any data or metadata. Therefore, they can be scaled horizontally. In this way, if any node fails, failover is easy, will not cause serious impact.

The blue shard node is the data node that actually stores data. Each shard node can be a single mongod instance or a replica set. When sharding is used, replica set is used to achieve high availability, so as to avoid service impact and data loss when a single node in the cluster fails. At the same time, you can further share the load through read/write splitting. Each database with sharding enabled has a default shard. Initially, the first chunk will be created there. The new data will be inserted to the shard node first.

In the figure, the purple config node stores metadata, including the data location, that is, the nodes where the data is located, and the cluster configuration information. The config node is also a common mongod ., A group of config nodes consists of three. These three config nodes are not a replica set. Their data synchronization is ensured by mongos performing two-phase commit. This is to avoid metadata synchronization caused by Replication delay. The config node achieves high availability to some extent. When one or two nodes fail, the config cluster becomes read-only. However, the entire sharding cluster can still read and write data normally. Data migration and automatic balancing are not supported.

What are the metadata stored in the config node? After connecting to mongos,

use config; show collections

The result is

settingsshardsdatabasescollectionschunksmongoschangelog

You can further view the data of these items. This config database is the data ing on the backend config node, providing a convenient portal to read metadata. What are these collections? Settings contains the sharding configuration information, such as the data block size, whether to enable automatic balancing. Shards stores information about backend shard nodes, including ip addresses and ports. Databases stores database information, whether to enable sharding, and shard by default. In collections, which collections enable sharding and which shard keys are used. The chunks contains the data location and the range of each chunk. Mongos contains mongos information. changelog is a capped collection that stores the latest 10 m metadata change records.

Mongodb sharding is also easy to build. A few simple steps can be completed.

Start several shard nodes first

mongod --shardsvr

Start three config nodes

mongod --configsvr

Start mongos

mongos --configdb=192.168.1.100, 192.168.1.101, 192.168.1.102

Here, the-shardsvr parameter only serves to change the default port to 27018, and-configsvr modifies the default port to 27019 and the default path to/data/configdb. In addition, it has no direct effect. In actual use, you can also specify the port and data path. In addition, the two parameters are used to mark the process, so that the process identity can be easily identified in the ps aux process list. -The configdb parameter is the config Node Address. If the default port is changed, add it here.

Then we add the data nodes to the cluster: run on mongos

use adminsh.addShard(’ [hostname]:[port]’)

If replicaSet is used, yes

use adminsh.addShard(’replicaSetName/,,’)

The next step is to enable sharding.

sh.enableSharding(dbname)sh.shardCollection(fullName, key, unique)

In this way, you can. It's easy. If there is data in the collection, the data is automatically balanced.

Previously, mongodb sharding divides data into chunks for management. Now let's see what chunk is. In mongodb sharding, chunk is the basic unit of data migration. Data in each node is divided into several chunks. A chunk is essentially a continuous interval of the shard key. A chunk is actually a logical division rather than a physical division. The sharding backend is a common mongod or replica set, and does not perform special processing on data because of sharding. A chunk does not actually store a page or a file, but is only reflected in the metadata of the config node. Mongodb's sharding policy is actually a range mode.

The range of the first chunk is from-∞ to 12000. The second is 12000 to 58000. And so on. For a collection Just configured as sharding, there is only one chunk at the beginning, ranging from-∞ to + ∞.

As the data grows, the data size exceeds the configured chunk size. The default value is 64 MB, and the chunk is split into two. Because chunk is a logical unit, splitting operations only involve metadata operations. The growth of data will split chunk more and more. At this time, the number of chunks on each shard will be unbalanced. At this time, a balancer component in mongos performs automatic balancing. Move the chunk from the shard node with the largest number of chunks to the smallest number of nodes.

Finally, the number of chunks on each shard node tends to be balanced. Of course, balance does not necessarily mean the data completely, because mobile data itself has a certain cost, and in order to avoid data migration back and forth in the morning in extreme circumstances, only when the difference in the number of chunks of two shard instances reaches a certain threshold. The default threshold value is 8. That is to say, by default, only when the data size difference between the two nodes reaches 64 M * 8 = 256 M. In this way, there is no need to insert a lot of data to the created sharding. Why is it still strange in a node. This is only because there is not enough data to be migrated.

During data migration, data can still be read and written without affecting availability. So how does mongodb do it? During data migration, data read/write operations are performed in the source data node first. After the migration is complete, synchronize the update operations during this period to the new node. At last, update the config node and mark that the data is already in the new location to complete the migration. Data Update must be locked only when the operation is performed during the last synchronization migration. In this case, the lock time should be minimized to greatly reduce the impact of data migration on the service.

The biggest difference between mongodb sharding and traditional sharding lies in the introduction of metadata. It seems that the complexity is increased and some additional storage is added, but the resulting flexibility is obvious. Traditional sharding is essentially a static ing of data, and all the difficulties in data migration come from this. After metadata is introduced, the static ing is changed to dynamic ing. Data migration is no longer difficult. This fundamentally solves the problem. On the other hand, the implementation of chunk with metadata reduces the implementation difficulty, and the back-end nodes can still use the original technology. At the same time, it is easier to deploy and migrate without changing the backend data. You only need to add mongos nodes and config nodes.

Let's talk about the data routing function. The primary function of mongos is to act as a data route, locate the data location, and merge the query results. Let's see how it works. If the query condition is the shard key, mongos can directly locate the chunk from the metadata and find the data from the target node.

If the query condition is the shard key range, because the chunk is divided by the shard key range, mongos can also find the location where the data corresponds to the chunk and merge the data returned by each node.

If the query condition is not any index, the original full collection traversal is still inevitable. However, it is distributed to all nodes. Therefore, it can still share the load.

If the query condition is an index but not a shard key, the query will be distributed to all nodes, but the index on each node is still valid.

If the chunk is sorted by the query shard key, because the chunk is the range of a shard key, the nodes where each chunk is located are queried in sequence, and no data is returned for sorting. If it is not sorted by the shard key, the ordering operation is performed on each node, and then the mongos? Sort by merging. Because it is a merge sort of sorted results, there is not much pressure on mongos, and the query result cursor will also be changed to the cursor on each node. You do not need to spit out all the data.

From the above we can see that for sharding clusters, shard key selection is crucial. The shard key is actually equivalent to the database's clustered index. Therefore, the principle of selecting the clustered index is similar to that of selecting the shard key. Similarly, once the shard key is set, it cannot be changed. Therefore, be cautious when selecting the shard key. The shard key is selected as follows.

First, the value of the shard key is fixed and will not be changed. Once this value is changed, it may be moved from one node to another, resulting in great overhead.

Second, the shard key must have sufficient discrimination. Because? What is a chunk? Shard key range, so the value of the shard key can only be in the same chunk. If the shard key has a large value, the chunk size exceeds the chunk size, and the chunk cannot be split and data is balanced. At the same time, like general database indexes, better discrimination can also improve query performance.

Third, the shard key must be random, not one-way growth. One-way growth of the shard key will cause the newly inserted data to be located in a chunk, resulting in a concentrated write pressure on a shard node. Therefore, it is best to avoid using _ id directly. The one-way growth value such as the timestamp is used as the shard key.

Mongodb sharding has many advantages, but it also has its limitations.

First, mongodb only provides sharding in the range mode. Although this mode can optimize the range query and sorting by shard key, it will also result in the use of one-way increasing values, write results in the set.

Second, after sharding is enabled, the uniqueness of other indexes except the shard key cannot be guaranteed. Even if it is set to unique, it is only guaranteed to be unique in each node. One way is to set the index? { : 1 ,? : 1 }. However, this does not necessarily meet the business logic requirements.

Third, you cannot directly use group () after sharding is enabled (). However, the map reduce function can be used as an alternative.

Fourth, although the data migration operation has little impact on read/write operations, this process must first Replace the data from the disk into the memory, which may damage the hot data cache. In addition, data migration also increases the I/O pressure, so you can consider disabling automatic balancing at ordinary times and performing it again when the pressure is low in the early morning.

At last, the metadata synchronization of the config node has a high requirement on clock accuracy. Once the clock error of each config node is large, locks will occur and cannot be changed, resulting in data concentration. Therefore, ntp clock synchronization is essential.

Here we will talk about the sharding cluster backup problem. Because the back-end data nodes are still common mongod or replica set, the backup is actually similar to the original one. Note that automatic balancing must be stopped before the backup to ensure that the sharding metadata does not change during the backup, and then the shard node and config node data can be backed up.

P.S. This article is the topic content I organized at thinkinlamp's third Database Conference in March.

Original article address: MongoDB Sharding mechanism analysis, thanks to the original author for sharing.