MongoDB to build a shard cluster

In MongoDB (version 3.2.9), sharding refers to the distribution of collection to different servers, where each server stores only a portion of the collection, and all of the services that serve the shards form a shard cluster. The server for the Shard cluster (sharded Clustered) is divided into three types: Router (mongos), Config server, and Shard (Replica Set or Standalone mongod). With fragmented clusters, you can store more data and handle larger workloads without the need for a powerful computer. The purpose of distributed database system is to divide the load into multiple servers and reduce the load of single-machine query.

One, configure the server

The config server stores the metadata for the shards, including the Block (chunk) list for each shard and the range of data that each chunk contains. The Routing service Area (Router) obtains the metadata of the shards from the config server and uses metadata to route read and write operations to the correct shards.

The metadata includes the list of chunks on every shard and the ranges that define the chunks. The MONGOs instances cache this data and use it to route read and write operations to the correct shards.

Config server has very few read and write operations, and config server stores the metadata of the shards in the Config database, modifying the config only if the metadata of the shards changes, such as Chunk Migration,chunk split. The data in the server. The data in config server is only read by MONGOs when MONGOs is first started or restarted, or when metadata for a shard changes. MONGOs is cached locally after the metadata for the Shard is read.

Config servers store the cluster ' s metadata in the Config database. The MONGOs instances cache this data and use it to route reads and writes to Shards. MongoDB only writes data to the config servers when the metadata changes, such as

• After a chunk migration, or
• After a chunk split.

MongoDB reads data from the config server in the following cases:

• A new MONGOs starts for the first time, or an existing MONGOs restarts.
• After cluster metadata, such as after a chunk migration.

In fact, config server is Mongod, just set the--CONFIGSVR option.

--CONFIGSVR Specifies Mongod as a config server

Two, MONGOs routing server

MONGOs provides a routing service for MongoDB, handles query requests sent from the application layer, locates the shards where the data resides, and combine the query results on the Shard to complete the distributed data query. From application's point of view, MONGOs's role is a MongoDB Instance, which hides the complex process of query and combine data from a shard.

Important parameters of MONGOs

--config <filename>,-f <filename> Specify parameters for MONGOs to run

--CONFIGDB specifies the config server list in the format: Config-svr:port,config-svr:port

--CHUNKSIZE Specifies the size of the data block, in megabytes, the default value is 64

--PORT Specifies the port number of the TCP that the MONGOs listens on, the default value is 27017

--LOGPATH specifies the path to the MONGOs logging, and by default, MongoDB renames the existing log file instead of rewriting it. By default, MongoDB would move any existing log file rather than overwrite it. To instead append to the log file, set the--logappend option.

Third, build a shard cluster

1,shard

Shards (Shard) are used to store data, either replica Set or standalone, because each Shard saves a portion of the data collection, and if Shard fails, collection becomes incomplete. In the product environment, each shard is a replica set.

2,config Server

Config server holds the mapping between each shard and the data, which shard the data is stored on, or what data is stored on each shard, and a doc can only be stored on one shard. The metadata for the Shard is extremely important, and you must enable Logging for Config server to ensure that the metadata is saved to disk. It is best to use 3 config servers, each of which should be located on a separate physical machine, preferably a geographically dispersed machine.

Create three config SERVER:CFG-SRV1,CFG-SVR2,CFG-SVR3 with configuration files located at:

• Cfg-svr1,c:\data\config\cfgsvr_1.conf
• Cfg-svr2,c:\data\config\cfgsvr_2.conf
• Cfg-svr3,c:\data\config\cfgsvr_3.conf

--config Server 1dbpath=c:\data\configlogpath=c:\data\config\cfgsvr_1.logjournal=trueport=50001configsvr= True--config Server 2dbpath=c:\data\configlogpath=c:\data\config\cfgsvr_2.logjournal=trueport=50002configsvr= True--config Server 3dbpath=c:\data\configlogpath=c:\data\config\cfgsvr_3.logjournal=trueport=50003configsvr= True

When you start config server, do not use the--replset parameter when you start the configuration servers, config server is not the replica set;--configsvr parameter that specifies Mongod as config server.

--config Server 1mongod-f C:\data\config\cfgsvr_1.conf

--config Server 2mongod-f C:\data\config\cfgsvr_2.conf

--config Server 3mongod-f C:\data\config\cfgsvr_3.conf

3,router
MONGOs is a routing server (Router), MONGOs requires the address List of config server, and--configdb specifies the list of config servers that can be accessed by Router. MONGOs do not save the data, you do not need to specify the DBPath parameter, MONGOs load the cluster data from config server at startup, you can start any number of MONGOs, each mongos use the same config server list.

Create a MONGOs on ROUTER-SVR1, store the configuration document in C:\data\mongos\cfg_mongos.conf, and use the--port parameter to specify the port that the MONGOs process listens on.

--mongos 1
logpath=c:\data\mongos\mongos_log.logport=60001configdb=cfg-svr1:50001,cfg-svr2:50002,cfg-svr2:50003

Start MONGOs

Mongos-f C:\data\mongos\cfg_mongos.conf

Four, Increase Shard

1, connect to MONGOs

MONGO--host ROUTER-SVR1--port 60001

View the state of a shard without any shard in the Shard cluster

Sh.status ()

2, Increase Shard

Each shard is used to store a shard of data, and the server that stores the data can be replica Set or standalone mongod.

Adding a replica Set shard to a shard cluster

Sh.addshard ("Replica_set_name/host:port")

Add a standalone Mongod to the Shard cluster

Sh.addshard ("Host:port")

3, enabling the database to enable Shard storage

Sh.enablesharding ("Database name")

4, enabling a collection in the database to enable Shard storage
Before you enable collection for shard storage, you must create a single-key or double-bond index on collection.

Db.collection_name.createIndex ({field:1}) sh.shardcollection ("Dbname.collection_name", {field:1})

5, insert into the collection, MongoDB will automatically manage shards

Db.collection_name.insert ({...})

Application connects MONGOs, writes or reads data, is routed by MONGOs to the appropriate shard, and this process is done automatically.

Reference Documentation:

sharded Cluster Administration

Sharding

Mongodb-sharding

MongoDB to build a shard cluster