MongoDB Shard Cluster deployed on CentOS 7 (MongoDB 3.2.1 version)

What is a shard?

High data volume and throughput of the database application will be more pressure on the performance of the single machine, large query volume will be a single CPU exhausted, large amount of data on the single-machine storage pressure, and eventually will exhaust the system's memory and transfer the pressure to disk IO.

MongoDB shards are a way to store data using multiple servers to support huge data stores and manipulate data. Shard Technology can meet the demand of large amount of MONGODB data, when a MongoDB server is not enough to store massive data or not enough to provide acceptable read and write throughput, we can divide the data on multiple servers, so that the database system can store and process more data.

MongoDB Shard Advantage

Sharding provides a way to cope with high throughput and large data volumes.

• Using sharding reduces the number of requests that each shard needs to process, so the cluster can increase its storage capacity and throughput by horizontally scaling. For example, when inserting a piece of data, the app only needs to access the shards that store the data.
• Using shards reduces the data stored for each shard.

The advantage of sharding is to provide a similar linear growth architecture, improve data availability, and improve the performance of large database query servers. Sharding technology is used when MongoDB single-point database server storage becomes a bottleneck, the performance of a single-point database server becomes a bottleneck, or you need to deploy large applications to take full advantage of memory.

The MONGODB shard structure looks like this:

MongoDB Shard Cluster composition

The MongoDB shard cluster consists of the following three main components:

• Shard: A shard server for storing actual blocks of data, a Shard server role in a real-world production environment can be made up of several servers, one replica set, to prevent a single point of failure of the host.
• Config server: Configures servers that store configuration information for the entire shard cluster, including chunk information.
• Routers: Front-end routing, which the client accesses and makes the entire cluster look like a single database,
  Front-end applications can be used transparently.

The composition of the MongoDB shard cluster is as follows:

Deploying a MongoDB Shard cluster

Here's how to deploy a simple fabric MongoDB shard cluster on a physical server.

物理服务器IP地址：192.168.10.154/24MongoDB软件包：mongodb-linux-x86_64-3.2.1.tgz具体配置如下：- 1台路由实例（端口27017）- 1台配置实例（端口37017）- 2台Shard实例（端口47017,47018）

Deploy a MongoDB shard cluster with a simple structure as shown in the following:

First, install the MongoDB 3.21, install the support package

yum install openssl-devel -y

2. Unzip the MongoDB package

tar zxvf mongodb-linux-x86_64-3.2.1.tgz -C /opt/

3, Mobile MongoDB extracted folder into the/usr/local/directory and renamed to MongoDB

cd /opt/mv mongodb-linux-x86_64-3.2.1/ /usr/local/mongodb

4, create data storage directory and log storage location

mkdir -p /data/mongodb/mongodb{1,2,3,4}   //创建数据存储目录mkdir /data/mongodb/logs    //创建日志存放位置touch /data/mongodb/logs/mongodb{1,2,3,4}.log    //日志文件chmod -R 777 /data/mongodb/logs/*.log    //给日志文件加权限才可往里面写入

5, adjust the shell resources settings to prevent excessive access to mongodb error and unable to connect MongoDB instance

ulimit -n 25000   //同一时刻最多开启文件数ulimit -u 25000   //同一时刻最多开启程序数

II. Deployment Configuration server 1, creating a mongodb1.conf configuration file

cd /usr/local/mongodb/bin/vim mongodb1.conf

port=37017          //端口号dbpath=/data/mongodb/mongodb1    //数据存放位置logpath=/data/mongodb/logs/mongodb1.log   //日志存放位置logappend=true   //使用追加方式写日志fork=true    //后台运行maxConns=5000    //最大同时连接数，默认2000storageEngine=mmapv1   //指定存储引擎为内存映射文件configsvr=true   //指定为配置服务器

2. Start the configuration server

mongod -f /usr/local/mongodb/bin/mongodb1.conf

Iii. Deploy Shard server 1, replicate configuration server configuration files and edit modifications

cd /usr/local/mongodb/bin/cp -p mongodb1.conf mongodb2.confcp -p mongodb1.conf mongodb3.confvim mongodb2.conf

port=47017    //端口号dbpath=/data/mongodb/mongodb2    //数据存放位置logpath=/data/mongodb/logs/mongodb2.log   //日志存放位置logappend=truefork=truemaxConns=5000storageEngine=mmapv1shardsvr=true   //指定为分片服务器

vim mongodb3.conf

port=47018    //端口号dbpath=/data/mongodb/mongodb3    //数据存放位置logpath=/data/mongodb/logs/mongodb3.log   //日志存放位置logappend=truefork=truemaxConns=5000storageEngine=mmapv1shardsvr=true     //指定为分片服务器

2. Start two shard servers

mongod -f mongodb2.confmongod -f mongodb3.conf

Iv. starting the routing server

The./mongos--help command allows you to view information about the parameters of the startup route. Chunksize is a block size, default is 200MB, for testing purposes here, set the value to 1.

./mongos --port 27017 --fork --logpath=/usr/local/mongodb/bin/route.log --configdb 192.168.10.154:37017 --chunkSize 1

V. Enable the Shard server

After connecting to the routing instance, you can view the Shard status information through the Sh.status () command

mongo          //进入路由实例

mongos> sh.status()      //shards下为空，没有分片服务器

mongos> sh.addShard("192.168.10.154:47017")   //添加分片服务器mongos> sh.addShard("192.168.10.154:47018")   //添加分片服务器mongos> sh.status()    //再次查看便有分片服务器

VI. Realization of Shard function

After adding two shard servers, the databases and collections have not yet been enabled for sharding, where I first create a XXY database and create a collection

mongos> use xxymongos> db.users.insert({"id":1,"name":"zhangsan"})mongos> show dbs

Can write multiple messages with circular statements

mongos> for(var i=2;i<=10000;i++)db.users.insert({"id":2,"name":"zhangsan"+i})mongos> show collections   mongos> db.users.find().limit(5)  //查看users集合里的前五条数据

Use the Sh.status () command to view Shard state information. A value of "partitioned" is false to indicate that the database is not fragmented.

mongos> sh.status()          //查看数据库分片信息

Use the sh.enablesharding ("XXY") command to enable XXY database sharding. Check again to see that the "partitioned" value is true

mongos> sh.enableSharding("xxy")      //启用数据库分片

Create an index for the users table, and then use the Sh.shardcollection ("Xxy.users", {"id": 1}) command to shard the Users collection

mongos> db.users.createIndex({"id":1})     //对users表创建索引mongos> sh.shardCollection("kgc.users",{"id":1})  //表分片mongos> sh.status()

At this point you can see that the chunks is evenly distributed over two shards

MongoDB Shard Cluster deployed on CentOS 7 (MongoDB 3.2.1 version)