A summary of the fragmentation mechanism of MONGODB learning notes

Fragmentation is a way to mongodb data horizontally, storing data evenly in Shard server clusters by selecting the right slice key.

The fragmented component consists of the Shard server cluster, the config server, and the MONGOs process. As shown in the following illustration:

Config server to save fragmentation-related metadata, that is, which Shard servers, and which Chunk,chunk are located in which Shard server.

MONGOs is primarily responsible for routing, forwarding client requests to the corresponding Shard server.

Real data is stored on the Shard server, shard server can be a mongod, or it can be a replica set.

Config server

Config server is a special Mongod instance that stores the metadata for the Shard cluster. One config server is prone to single points of failure, and 3 config servers are officially recommended to ensure data security and reduce data synchronization overhead among multiple config servers. As long as any one of the config servers is unavailable, the config server cluster becomes read-only and, if all is dead, the read and write requests before the reboot are lost. You can modify the name and quantity of the config server without stopping.

Config server saves the metadata through config database, and MONGOs caches the metadata. The partition-related collections in config database are:

Config.changelog record meta data changes;

Config.chunks records the information of each chunk, including the Shard of Chunk, and the Min and Max values of chunk;

Config.lockpings and Config.locks Store the distributed lock related information, when the equalizer needs to acquire the lock first;

Config.mongos stores MONGOs instance information, MONGOs instances have heartbeat detection every 30 seconds;

Config.settings fragment related configuration, such as chunk size, whether equalizer is enabled, etc.

Config.shards records the corresponding machine for each shard, and if Shard is a replica set, the name of the corresponding replica set.

Chunk

The data in Shard is divided into multiple chunk, the default size of each chunk is 64M, and if chunk is larger than 64M, MONGOs attempts to split it, resulting in inconsistent shard numbers for each chunk. MONGOs will migrate these uneven fragments.

If you find that the 64M migration process is too long, you can reduce the block size to increase migration speed, when you modify the block size, the existing blocks are not split immediately, but at the next split time.

Equalization Device

Because migration is cost-MongoDB is not immediately migrated when chunk numbers are found to be inconsistent, there is a migration thresholds concept exists, migration thresholds values are as follows:

Less than 20	2
20~79	4
>=80	8

The migration process is as follows:

(1) MONGOs send Movechunk command to source fragment

(2) The source fragment performs the internal movechunk operation, then writes to the chunk writes to the source fragment

(3) Purpose fragment from source fragment copy data, one chunk

(4) After the completion of the last chunk, followed by a synchronous operation, the migration process to synchronize the changes to the purpose of fragmentation

(5) After the synchronization completes, the purpose fragment connects to the Config database, updates the metadata, modifies the migration block corresponding shard information

(6) After the success of the update metadata, delete chunk data on the source fragment

Equalizer is divided into manual equalization and automatic Equalization, after Setbalancerstate (false) set, the equalizer does not enter the equalization process, you can view the state of the equalizer in the locks table. When a new fragment is added to the cluster, the equalizer attempts to write data for the fragment, triggering a series of migration processes that affect application performance, generally closing the equalizer when adding a fragment, or controlling the start time range of the equalizer.