MongoDB Shard cluster (enables Shard service enablement, Shard service Management, single point of failure simulation)

MongoDB Shard Overview 1, what is a shard

• High data volume and throughput of the database application will be more pressure on the performance of the single machine, large query volume will be a single CPU exhausted, large amount of data on the single-machine storage pressure, and eventually will exhaust the system's memory and transfer the pressure to disk IO.
• MongoDB shards are a way to store data using multiple servers to support huge data stores and manipulate data. Sharding technology can meet the demand of large amount of MONGODB data volume, when a server is not enough storage, you can add another shard server, so that the database system can store more data

2. MongoDB Shard Advantage

• Using sharding reduces the number of requests that each shard needs to process, so by horizontally scaling the cluster can improve its storage capacity and throughput, for example, when inserting a piece of data, the app only needs to access the shards that store the data.
• Using shards reduces the data stored for each shard.
• The advantage lies in providing a similar linear growth architecture, improving data availability, improving the performance of large database query servers, and using sharding technology when MongoDB single-point database server storage becomes a bottleneck, the performance of a single-point database server becomes a bottleneck, or you need to deploy large applications to take advantage of memory.

3. The composition of MongoDB shard cluster

• Shard: A shard server for storing actual blocks of data, a shard server role in a real-world production environment can be made up of several servers, one replica set, to prevent a single point of failure of the host.
• Config server: Configures servers that store configuration information for the entire shard cluster, including chunk information.
• Routers: Front-end routing, where the client is connected and makes the entire cluster look like a single database, and front-end applications can be used transparently.

MongoDB Shard Cluster Deployment cluster environment

Shard Server	corresponding configuration file	Service Port Settings
Configure the server	Mongod1.conf	37017
Shard server 1	Mongod2.conf	47017
Shard server 2	Mongod3.conf	47018
Shard Server 3	Mongod4.conf	47019
Routing server	Mongod.conf	27017

Installing MongoDB3.2

• Note: The MongoDB3.2 version chosen here is due to increased security after the MongoDB3.4 version, the following Shard server needs to be set up for Shard management in a replication set, where the latest version of the Shard Cluster management-related content will be supplemented in order to simplify the operation by selecting the MongoDB2.3 version.

• MongoDB2.3 version can be downloaded at MongoDB official website: https://www.mongodb.com/

• Install the Environment pack

Yum Install Openssl-devel-y

• Unpack packages, create working directory and log files

Tar zxvf mongdb-linux-x86_64-3.2.1.tgz-c/opt
MV MONGODB-LINUX-X86_64-3.2.1//usr/local/mongodb
Mkdir-p/data/mongodb/mongodb{1,2,3,4,5}
Mkdir/data/mongodb/logs
Touch/data/mongodb/logs/mongodb{1,2,3,4,5}.log
Chmod-r 777/data/mongodb/logs/*.log

• For system memory and file optimization

Ulimit-n 25000
Ulimit-u 25000

• Create each node configuration file

• Configure the server

Vim/etc/mongod1.conf

port=37017     //服务端口号dbpath=/data/mongodb/mongodb1       //工作目录logpath=/data/mongodb/logs/mongodb1.log  //日志文件位置logappend=truefork=truemaxConns=5000storageEngine=mmapv1configsvr=true      //设置为配置服务器

• allocating memory from other nodes when a node is low on memory

Sysctl-w vm.zone_reclaim_mode=0
echo Never >/sys/kernel/mm/transparent_hugepage/enabled
echo Never >/sys/kernel/mm/transparent_hugepage/defrag

• Create MongoDB Common command soft connection, easy to system identification, can be used directly

Ln-s/usr/local/mongodb/bin/mongo/usr/bin/mongo
Ln-s/usr/local/mongodb/bin/mongod/usr/bin/mongod

• Create a configuration file for the Shard server, including mongodb3.conf and mongodb4.conf

Vim/etc/mongodb2.conf

port=47017    //其他两个分片服务器端口分别为47018和47019dbpath=/data/mongodb/mongodb2      //工作目录也要更换logpath=/data/mongodb/logs/mongodb2.log   //日志文件也要更改logappend=truefork=truemaxConns=5000storageEngine=mmapv1shardsvr=true    //设置为分片服务器功能

• Start the server

Mongod-f mongodb1.conf
Mongod-f mongodb2.conf
Mongod-f mongodb3.conf
Mongod-f mongodb4.conf

• Start the routing server
• The routing server does not need to create working directories and log files, which can be specified at startup by command

./mongos --port 27017 --fork --logpath=/usr/local/mongodb/bin/route.log --configdb 192.168.144.113:37017 --chunkSize 1//客户端通过27017端口访问，工作模式为fork，日志文件位置及配置文件位置，交给配置服务器的37017去处理调度分片服务器

• When the Enable routing server appears, the following indicates a successful start

2018-07-16T21:54:55.081+0800 W SHARDING [main] Running a sharded cluster with fewer than 3 config servers should only be done for testing purposes and is not recommended for production.about to fork child process, waiting until server is ready for connections.forked process: 3058child process started successfully, parent exiting

Routing server Add Shard server

• To be able to make way for the server to locate the Shard server by configuring the server, you need to add the Shard server information to the routing server
• Enter the routing server

Mongo

mongos> show dbsmongos> sh.status()    //此时查看shards下为空，没有分片服务器mongos> sh.addShard("192.168.144.113:47017")    //添加分片服务器mongos> sh.addShard("192.168.144.113:47018")mongos> sh.status()     //再次查看shards下有了分片服务器

Enabling the Shard Store feature

• When the routing server can find the Shard server and cannot store the data on a different Shard server, this feature needs to be turned on.

• First look at the Shard storage feature is not open.

mongos> use kgc   //创建数据库mongos> for(var i=1;i<=10000;i++)db.users.insert({"id":i,"name":"jack"+i})  //创建一万条数据mongos> sh.status()          //查看数据库分片信息databases:    {  "_id" : "kgc",  "primary" : "shard0000",  "partitioned" : false }  //当看到false时表示分片存储功能未开启

• To take a data shard store for a collection in a database, you first need to create an index of the numerical control inside the collection, and then specify to Shard the collection in the database. The operation is as follows:

mongos> db.users.createIndex({"id":1})   //对users集合创建索引mongos> sh.shardCollection("kgc.users",{"id":1})  //集合分片mongos> sh.status()

• When the status is viewed, The Shard store is complete.

Shard Service Management

• To make it easier to see and remember, we can add tags to different shard servers.
• When the amount of data is increasing, MongoDB's shard storage function can realize on-line capacity expansion, that is, to increase the Shard server, so that the data can be automatically averaged shard storage, avoid single point of failure and maintenance without downtime.

Add tags

• After the Shard server boot and shard storage functionality is complete

mongos> sh.addShardTag("shard0000","first")mongos> sh.addShardTag("shard0001","second")  //添加标签mongos> sh.removeShardTag("shard0000","sales01")     //删除标签

• When seen as shown, indicates that the label was added successfully.

Link Configuration Server

• Since all operations on the Shard server through the routing server are done through the configuration server, you can go to the configuration server to view it.

MONGO--port 37017

configsvr> use config    //配置数据库configsvr> show collections   ....collections  //集合信息chunks       //分片节点信息databases    //数据库信息....configsvr> db.chunks.findOne() configsvr> db.collections.find()configsvr> db.databases.find()

Adding a shard server

• Dynamically add a shard server that automatically allows for average shard storage

• First start the Shard server you need to add

Mongod-f mongodb4.conf
MONGO//Enter the routing server

mongos> sh.addShard("192.168.144.113:47019")     //添加分片服务器mongos> sh.status()    //查看状态chunks:                shard0000   4                shard0001   5                shard0002   2

Delete a shard server

• When a shard server is deleted, that is, when a single point of failure is simulated, the configuration server will reassign the data in the database and store it on average on the other Shard servers without losing data.

mongos> use adminmongos> db.runCommand({"removeshard":"192.168.144.113:47019"})

• Analog single point of failure, results

MongoDB Shard cluster (enables Shard service enablement, Shard service Management, single point of failure simulation)