Deploying a MongoDB shard cluster on CentOS7

Overview

MongoDB shards are a way to store data using multiple servers to support huge data stores and manipulate data. Shard Technology can meet the demand of a large number of MONGODB data volume, when a MongoDB server is not enough to store massive amounts of data or insufficient to provide acceptable read and write throughput, we can split the data on multiple servers, so that the database system can store and process more data.

The composition of a MongoDB shard cluster

• Shard: A shard server for storing actual blocks of data, a Shard server role in a real-world production environment can be made up of several servers, one replica set, to prevent a single point of failure of the host.
• Config server: Configures servers that store configuration information for the entire shard cluster, including chunk information.
• Routers: Routers, which the client accesses, and makes the entire cluster look like a single database, the front-end application can be used transparently.

Deploying a MongoDB Shard cluster

Lab Environment:

• 1 Routing instances (port 27017)
• 1 Configuration instances (port 37017)
• 2 Shard instances (port 47017,47018)

1. Installing MongoDB3.2

[[email protected] ~]# yum install openssl-devel -y[[email protected] tomcat]# tar zxvf mongodb-linux-x86_64-3.2.1.tgz -C /opt/

2. Creation of 4 instances

[[email protected] opt]# mkdir -p /data/mongodb/mongodb{1,2,3,4}[[email protected] mongodb]# mkdir logs[[email protected] mongodb]# touch logs/mongodb{1,2,3,4}.log[[email protected] logs]# chmod -R 777 *.log[[email protected] logs]# ulimit -n 25000   //最大进程数//[[email protected] logs]# ulimit -u 25000  //最大文件数//

3. Deploying the Configuration Server

[[email protected] bin]# vim mongodb1.confport=37017dbpath=/data/mongodb/mongodb1logpath=/data/mongodb/logs/mongodb1.loglogappend=truefork=truemaxConns=5000storageEngine=mmapv1configsvr=true~      

1) allocating memory from other nodes when a node is low on memory

[[email protected] bin]# sysctl -w vm.zone_reclaim_mode=0 //内核参数为0的话，那么系统会倾向于从其他节点分配内存//[[email protected] bin]# echo never > /sys/kernel/mm/transparent_hugepage/enabled[[email protected] bin]# echo never > /sys/kernel/mm/transparent_hugepage/defrag[[email protected] bin]# ln -s /usr/local/mongodb/bin/mongo /usr/bin/mongo[[email protected] bin]# ln -s /usr/local/mongodb/bin/mongod /usr/bin/mongod

2) Start the configuration server

[[email protected] bin]# mongod -f mongodb1.conf

4. Configuring the Shard Server

[[email protected] bin]# vim mongodb2.conf port=47017dbpath=/data/mongodb/mongodb2logpath=/data/mongodb/logs/mongodb2.loglogappend=truefork=truemaxConns=5000storageEngine=mmapv1shardsvr=true[[email protected] bin]# vim mongodb3.conf port=47018dbpath=/data/mongodb/mongodb3logpath=/data/mongodb/logs/mongodb3.loglogappend=truefork=truemaxConns=5000storageEngine=mmapv1shardsvr=true

1) Start MONGODB2, mongodb3

[[email protected] bin]#mongod -f mongodb2.conf[[email protected] bin]#mongod -f mongodb3.conf

5. Start the routing server

[[email protected] bin]# ./mongos --port 27017 --fork --logpath=/usr/local/mongodb/bin/route.log --configdb 192.168.126.141:37017 --chunkSize 1************显示*******************************************2018-07-18T09:21:53.507+0800 W SHARDING [main] Running a sharded cluster with fewer than 3 config servers should only be done for testing purposes and is not recommended for production.about to fork child process, waiting until server is ready for connections.forked process: 3580child process started successfully, parent exiting

6. Enable the Shard server

[[email protected] bin]# mongomongos> show dbsconfig  0.031GBmongos> sh.status()    //#shards下为空，没有分片服务器//mongos> sh.addShard("192.168.126.204:47017")mongos> sh.addShard("192.168.126.204:47018")mongos> sh.status()shards:    {  "_id" : "shard0000",  "host" : "192.168.126.141:47017" }    {  "_id" : "shard0001",  "host" : "192.168.126.141:47018" }

1) Create the KGC database, create the Users collection, and insert 100,000 data

mongos> use kgcmongos> db.createCollection(‘users‘)mongos> for(var i=1;i<=100000;i++)db.users.insert({"id":1,"name":"jack"+i})mongos> db.users.find(){ "_id" : ObjectId("5b4e9b580f25d0730817aea1"), "id" : 1, "name" : "jack1" }{ "_id" : ObjectId("5b4e9b580f25d0730817aea2"), "id" : 2, "name" : "jack1" }{ "_id" : ObjectId("5b4e9b580f25d0730817aea3"), "id" : 3, "name" : "jack1" }{ "_id" : ObjectId("5b4e9b580f25d0730817aea4"), "id" : 4, "name" : "jack1" }{ "_id" : ObjectId("5b4e9b590f25d0730817aea5"), "id" : 5, "name" : "jack1" }{ "_id" : ObjectId("5b4e9b590f25d0730817aea6"), "id" : 6, "name" : "jack1" }{ "_id" : ObjectId("5b4e9b590f25d0730817aea7"), "id" : 7, "name" : "jack1" }{ "_id" : ObjectId("5b4e9b590f25d0730817aea8"), "id" : 8, "name" : "jack1" }{ "_id" : ObjectId("5b4e9b590f25d0730817aea9"), "id" : 9, "name" : "jack1" }{ "_id" : ObjectId("5b4e9b590f25d0730817aeaa"), "id" : 10, "name" : "jack1" }..........//省略//

7. Implementing Sharding Functionality

[[email protected] bin]# ./mongoimport -d kgc -c users --file /opt/testdb.txtmongos> show dbsconfig  0.031GBkgc     0.078GBmongos> use kgcswitched to db kgcmongos> show tablessystem.indexesusersmongos> sh.status()    //查看数据库分片信息// databases:    {  "_id" : "kgc",  "primary" : "shard0000",  "partitioned" : false } //数据库尚未分片//

1) Enable database sharding

mongos> sh.enableSharding("kgc")   //启用数据库分片//mongos> sh.status()databases:    {  "_id" : "kgc",  "primary" : "shard0000",  "partitioned" : true }mongos> db.users.createIndex({"id":1})   //对users表创建索引//mongos> sh.shardCollection("kgc.users",{"id":1})  //表分片//mongos> sh.status() shards:    {  "_id" : "shard0000",  "host" : "192.168.126.141:47017" }    {  "_id" : "shard0001",  "host" : "192.168.126.141:47018" }

Deploying a MongoDB shard cluster on CentOS7