Deploying a MongoDB Shard cluster

Overview of deploying MongoDB Shard cluster shards

High data volume and throughput of the database application of the performance of a single machine will be more pressure, large query volume will be a single CPU exhausted, large amount of data on the single-machine storage pressure, and eventually will exhaust the system's memory and transfer the pressure to disk IO.

MongoDB shards are a way to store data using multiple servers to support huge data storage and manipulation of data. Sharding technology can meet the demand for a large number of MONGODB data volume, when a MongoDB server is not enough to store massive data or not enough to provide acceptable read and write throughput, we can divide the data on multiple servers, so that the database system can store and process more data.

MongoDB Shard Advantage

Sharding provides a way to cope with throughput and large data volumes.

1, so that the Shard reduces the number of requests per shard to process, so through horizontal scaling, the cluster can improve its storage capacity and throughput. For example, when inserting a piece of data, the app only needs to access the shards that store the data.

2. Using shards reduces the data stored for each shard.

As shown, suppose that a collection has 1TB of data, through the Shard can be fragmented into four shard, when the corresponding data query, will go to the corresponding Shard server query, can improve the availability of data.

MongoDB Shard Cluster composition

There are three main components of the MongoDB shard cluster:

1, Shard: The Shard server, used to store the actual data block, a Shard server role in the actual production environment can be made up of a replica set of several servers (Replica set), to prevent the host single point of failure.

2. config server: Configures the servers to store configuration information for the entire shard cluster, including chunk information.

3, Routers: front-end routing, the Client access, and make the entire cluster look like a single database, front-end applications can be used transparently.

Experimental environment

The experiment here is to build mongodb shards on the same server, but after MongoDB3.4 the Shard default is to build the MongoDB service set, for the sake of convenience, we use the MongoDB version is 3.2.

Components	Port number
Routing instances	27017
Configuration instance	37017
Shard1 instances	47017
Shard2 instances	47018

Shard Cluster Deployment 1, deploying a configuration server

Install the dependent package and decompression software into the specified directory.

[[email protected] ~]# yum install openssl-devel -y   #安装依赖包[[email protected] ~]# tar xfvz mongodb-linux-x86_64-3.2.1.tgz -C /opt/[[email protected] ~]# cd /opt/[[email protected] opt]# mv mongodb-linux-x86_64-3.2.1/ /usr/local/mongodb#将软件包重命名，方便下面使用

The

Creates the data directories, log directories, and log files that are required for the instance that will be created first.

[[email protected] opt]# mkdir-p/data/mongodb/mongodb{1,2,3,4} #创建实例的数据存储目录 [[email protected] opt]# mkdir /data/mongodb/logs# creating the log directory for the instance [[email protected] opt]# touch/data/mongodb/logs/mongodb{1,2,3,4}.log# Creating the log file for the instance [[email protected] opt]# chmod-r 777/data/mongodb/logs/*.log# Modify log permissions [[email protected] opt]# ulimit-n 25000 #将可以打开的最大进程数修改为25000 [[email protected] opt]# ulimit-u 25000# Change the maximum number of files that can be opened to 25000[[email protected] opt]# Sysctl-w vm.zone_reclaim_mode=0# When a node is low on available memory, if it is 0, then the system tends to allocate memory from other nodes. If it is 1, then the system tends to reclaim the cache memory from the local node most of the time. [[email protected] opt]# echo never >/sys/kernel/mm/transparent_hugepage/enabled[[email protected] opt ]# echo never >/sys/kernel/mm/transparent_hugepage/defrag# when the node is running out of memory, choose to close the large page that accounts for more memory [[email protected] opt]# Export path= $PATH:/usr/local/mongodb/bin/#将MongoDB路径添加到环境变量中 [[email protected] bin]# vim mongodb1.conf# Create a configuration file for the configuration server Port=37017dbpath=/data/mongodb/mongodb1logpath=/data/mongodb/logs/mongodb1.logloGappend=truefork=truemaxconns=5000storageengine=mmapv1configsvr=true #设置为配置服务器 [[email protected] mongodb]# Mongod-f/usr/local/mongodb/bin/mongodb1.conf #启动配置服务器进程

2. Configure the Shard server

Edit the mongodb2.conf configuration file, Port 47017, set to Shardsvr=true, same edit mongodb3.conf, Port 47018,shardsvr=true, and start at the same time.

[[email protected] mongodb]# cd /usr/local/mongodb/bin/[[email protected] bin]# cp -p mongodb1.conf mongodb2.conf [[email protected] bin]# vim mongodb2.conf port=47017      #设置端口dbpath=/data/mongodb/mongodb2    #数据存储目录要修改logpath=/data/mongodb/logs/mongodb2.log    #日志文件也要修改logappend=truefork=truemaxConns=5000storageEngine=mmapv1shardsvr=true   #设置为分片服务器[[email protected] bin]# cp -p mongodb2.conf mongodb3.conf [[email protected] bin]# vim mongodb3.conf port=47018     #设置端口dbpath=/data/mongodb/mongodb3     #数据存储目录要修改logpath=/data/mongodb/logs/mongodb3.log     #日志文件也要修改logappend=truefork=truemaxConns=5000storageEngine=mmapv1shardsvr=true

The two shard servers are started after the modifications are complete.

3. Start the routing server

[[email protected] bin]# ./mongos --port 27017 --fork --logpath=/usr/local/mongodb/bin/route.log --configdb 192.168.58.131:37017 --chunkSize 1#这条命令意思就是凡事访问mongodb 27017端口的，都会转到 192.168.58.131:37017上面[[email protected] bin]# ./mongos --port 27017 --fork --logpath=/usr/local/mongodb/bin/route.log --configdb 192.168.58.131:37017 --chunkSize 12018-07-17T17:20:58.990+0800 W SHARDING [main] Running a sharded cluster with fewer than 3 config servers should only be done for testing purposes and is not recommended for production.about to fork child process, waiting until server is ready for connections.forked process: 6430child process started successfully, parent exiting

[[email protected] ~]# mongoMongoDB shell version: 3.2.1connecting to: testServer has startup warnings: 2018-07-17T17:20:59.120+0800 I CONTROL  [main] ** WARNING: You are running this process as the root user, which is not recommended.2018-07-17T17:20:59.120+0800 I CONTROL  [main] mongos> sh.status()--- Sharding Status ---   sharding version: {    "_id" : 1,    "minCompatibleVersion" : 5,    "currentVersion" : 6,    "clusterId" : ObjectId("5b4db4fdf19287c74d1a4081")}  shards:  active mongoses:    "3.2.1" : 1  balancer:    Currently enabled:  yes    Currently running:  no    Failed balancer rounds in last 5 attempts:  0    Migration Results for the last 24 hours:         No recent migrations  databases:#可以看到shards下面为空，所以我们需要将两个分片服务器添加进去

mongos> sh.addShard("192.168.58.131:47017")#将47017端口的分片服务器添加进去{ "shardAdded" : "shard0000", "ok" : 1 }mongos> sh.addShard("192.168.58.131:47018")#将47018端口的分片服务器添加进去{ "shardAdded" : "shard0001", "ok" : 1 }

To test to see if the Shard function is possible, we create a collection that inserts 50,000 data.

mongos> use testswitched to db testmongos> for(var i=1;i<=50000;i++) db.users.insert({"id":i,"name":"tom"+i})#利用for循环往collection中插入50000条数据mongos> db.users.find().limit(10)#查看collection中的前10条数据，表示插入成功。

mongos> sh.status ()---sharding status---sharding version: {"_id": 1, "mincompatibleversion": 5, "Curre Ntversion ": 6," Clusterid ": ObjectId (" 5b4db4fdf19287c74d1a4081 ")} shards: {" _id ":" shard0000 "," host ":" 192. " 168.58.131:47017 "} {" _id ":" shard0001 "," host ":" 192.168.58.131:47018 "} Active Mongoses:" 3.2.1 ": 1 Balanc er:currently Enabled:yes currently Running:no Failed balancer rounds in last 5 attempts:0 migration Res Ults for the last Hours:no recent migrations databases: {"_id": "Test", "PRIMARY": "shard0000", "par Titioned ": false} #可以看到partitioned: false, stating that there is no shard.  Mongos> sh.enablesharding ("test") {"OK": 1} #启用分片功能mongos > Sh.status () {"_id": "Test", "PRIMARY": "shard0000",            "Partitioned": True}mongos> db.users.createIndex ({"id": 1}) {"Raw": {"192.168.58.131:47017": { "Createdcollectionautomatically": false, "Numindexesbefore": 1, "numIndexesafter ": 2," OK ": 1}," OK ": 1} #创建索引, convenient shard mongos> sh.shardcollection (" Test.users ", {" I D ": 1}) {" collectionsharded ":" Test.users "," OK ": 1} #执行分片, can see Shard success 1-20165, write to shard0001, 20166-50000 write to shard0000

Deploying a MongoDB Shard cluster