MongoDB Shard Build (Single instance)

One: introduction of Shards

It is a kind of database cluster system that expands the massive data horizontally, the data table is stored on each node of sharding, and the user can easily construct a distributed MongoDB cluster by simple configuration.

The data chunking of MongoDB is called Chunk. Each chunk is a contiguous data record in the Collection, usually with a maximum size of 200MB, and a new chunk is generated beyond that.

To build a MongoDB sharding Cluster, you need three roles:

Shard Server

That is, the shards that store the actual data, each shard can be an Mongod instance, or a set of replica set that consists of Mongod instances. In order to implement each shard internal Auto-failover,mongodb the official recommendation for each shard is a set of replica set.

Config Server

In order to store a particular collection in multiple Shard, you need to specify a shard key for the collection, for example {age:1}, and Shard key determines which chunk the record belongs to. Config servers is used to store the configuration information for all shard nodes, the Shard key range for each chunk, chunk distribution in each Shard, and collection configuration information for all DB and sharding in the cluster.

Route Process

This is a front-end route that the client accesses, then asks config servers which shard to query or save the record, and then connect the corresponding shard to operate, and finally return the results to the client. The client simply sends the query or update request that was originally sent to mongod to routing Process without worrying about which shard the record is stored on.

Below we build a simple sharding Cluster on the same physical machine:

The architecture diagram is as follows:

Second: Experimental environment

Simulate a configuration server on the same machine, one MONGOs server, 2 shards (each shard is a single instance)

Three: Experimental steps
3.1 Download Unzip MongoDB

It should be downloaded on each server to unzip MongoDB. Since this is done on the same machine, it needs to be decompressed once.

TAR-XVF mongodb-linux-x86_64-2.6.9.tgz

# to facilitate management, move the installation files to/data

[ROOT@PC download]# MV Mongodb-linux-x86_64-2.6.9/data/mongodb

3.2 Creating a related directory

Location and log files for creating a database for MongoDB

[root@pc mongodb]# cd/data/mongodb/
[root@pc mongodb]# mkdir data
[root@pc mongodb]# touch Logs
[root@pc mongodb]# ls-ltr Total
-rw-r--r--1 1046 1046  1359 Mar 22:49 README
-rw-r--r--1 1046 1046 34520 Ma R gnu-agpl-3.0
-rw-r--r--1 1046 1046 17793 Mar 22:49 22:49 third-party-notices
drwxr-xr-x 2 root root  4096 Jul 13:26 bin
drwxr-xr-x 2 root root  4096 Jul 13:34 data
-rw-r--r--1 root root     0 Jul 15 13 : Logs
 
#建配置信息服务器数据文件目录
[root@pc download]# mkdir-p/dbs/config
 
#建分片目录
mkdir-p/dbs/shard1
mkdir-p/dbs/shard2
 
#建日志目录
mkdir/logs
 
#建日志文件
touch/logs/config.log 
Touch Mongos.log 
Touch Shard1.log 

3.3 Configuring Path

[Root@pc dbs]# Vi/root/.bash_profile

Add at the end of the path line:

:/data/mongodb/bin

[Root@pc dbs]# Source/root/.bash_profile

3.4 Starting the configuration server

The configuration server needs to be started first, and the MONGOs server needs to use the configuration information above. The configuration server starts just like a normal mongod.

Configuring the server does not require much space and resources (200MB of actual data is approximately 1KB of configuration space)

[root@pc dbs]# mongod--dbpath=/dbs/config--logpath=/logs/config.log--fork--port 1000

About-to-fork child process, waiting until server was ready for connections.

Forked process:5055

Child process started successfully, parent exiting

[Root@pc ~]# Ps-ef | grep MONGO

Root 9691 1 3 22:31? 00:00:00 mongod--dbpath=/dbs/config--logpath=/logs/config.log--fork--port 1000

3.5 Starting the MONGOs server

[root@pc ~]# mongos--port 1001--logpath=/logs/mongos.log--configdb=192.168.6.51:1000--fork
2015-07-16T22 : 32:36.776-0700 warning:running with 1 config server should are done only for testing purposes and are not recommended for Production about
to fork child process, waiting until server was ready for connections.
Forked process:9716 Child
process started successfully, parent exiting

--CONFIGDB here is the IP and port number where the configuration server is located.

When configuring--CONIGDB, the IP address cannot be populated with localhost or 127.0.0.1 the following error message is returned when adding shards:

1.    {
2.            " OK ": 0,
3.            " ErrMsg ":" Can ' t use localhost as a shard since all shards need to communicate. Either use all shards and configdbs on localhost or all in actual IPs  host:192.168.6.51:1000 islocalhost:0 "
4.
  }

3.6 starting a shard

#分片1

[root@pc ~]# mongod--dbpath=/dbs/shard1--logpath=/logs/shard1.log--fork--port 1002

About-to-fork child process, waiting until server was ready for connections.

Forked process:10583

Child process started successfully, parent exiting

#分片2

[root@pc ~]# mongod--dbpath=/dbs/shard2--logpath=/logs/shard2.log--fork--port 1003

About-to-fork child process, waiting until server was ready for connections.

Forked process:10659

Child process started successfully, parent exiting

3.7 Adding shards

#注意: Adding shards and sharding data is done on the MONGOs server.

Log on to the MONGOs server,

[Root@pc mongodb]# MONGO Localhost:1001/admin

MongoDB Shell version:2.6.9

Connecting To:localhost:1001/admin

#添加分片

Mongos> Db.runcommand ({addshard: "localhost:1002", allowlocal:true});

{"shardadded": "shard0000", "OK": 1}

Mongos> Db.runcommand ({addshard: "localhost:1003", allowlocal:true});

{"shardadded": "shard0001", "OK": 1}

#查看所有的片

mongos> use config;
Switched to DB Config

Mongos> Db.shards.find ();

{"_id": "shard0000", "host": "Localhost:1002"}

{"_id": "shard0001", "host": "Localhost:1003"}

3.8 Slicing Data

By default, every piece of data stored is not fragmented, and fragmentation is required on the granularity of the database and collection.

Here take dba.b as an example

mongos> use DBA;

Switched to DB DBA

Mongos> db.createcollection ("B");

{"OK": 1}

3.8.1 The Shard function of the Open library

mongos> use admin;

Switched to DB admin

Mongos> Db.runcommand ({"enablesharding": "DBA"});

{"OK": 1}

3.8.2 The sharding function of the Open table

Mongos>  Db.runcommand ({"Shardcollection": "Dba.b", "key": {"id": 1}})
{"collectionsharded": "dba.b", "OK" : 1}
Note: You need to switch to the Admin library to execute the command. TAB
key: The key above is the so-called Chip key (Shard key). MongoDB does not allow inserting documents without a slice key. However, the different types of chip keys are allowed in various documents, and MongoDB has a sort of different type:
 
mongos> use config;
Switched to DB config
mongos> db.databases.find ();
{"_id": "admin", "partitioned": false, "PRIMARY": "Config"}
{"_id": "DBA", "partitioned": true, "PRIMARY": "shard0000"}
 
You can see that the DBA is partitioned by this library.
 
mongos> db.chunks.find ();
{"_id": "Dba.b-id_minkey", "Lastmod": Timestamp (1, 0), "Lastmodepoch": ObjectId ("55a8c850fe175819f8525593"), "ns": "D Ba.b "," min ": {" id ": {" $minKey ": 1}}," Max ": {" id ": {" $maxKey ": 1}}," Shard ":" shard0000 "}
 
Chunks: Understanding M The key to Ongodb fragmentation mechanism is to understand chunks. MongoDB does not store an interval on a shard, but each shard contains multiple intervals, each of which is a block.

3.9 Shard Test

mongos
> Use DBA;
Switched to DB DBA mongos> for (i=0;i<100000;i++) {Db.b.insert ({"id": I, "Name": "Baidandan", "date": New Date ()});}
Writeresult ({"ninserted": 1}) mongos> use config;
Switched to DB config mongos> db.chunks.find (); {"_id": "Dba.b-id_minkey", "Lastmod": Timestamp (2, 1), "Lastmodepoch": ObjectId ("55a8c850fe175819f8525593"), "ns": "D Ba.b "," min ": {" id ": {" $minKey ": 1}}," Max ": {" id ": 0}," Shard ":" shard0000 "} {" _id ":" dba.b-id_0.0 "," last  MoD ": Timestamp (1, 3)," Lastmodepoch ": ObjectId (" 55a8c850fe175819f8525593 ")," ns ":" Dba.b "," min ": {" id ": 0}," Max " : {"id": 5562}, "Shard": "shard0000"} {"_id": "dba.b-id_5562.0", "Lastmod": Timestamp (2, 0), "Lastmodepoch": OBJ Ectid ("55a8c850fe175819f8525593"), "ns": "Dba.b", "min": {"id": 5562}, "Max": {"id": {"$maxKey": 1}}, "Shard" : "Shard0001"} 

We can see that there are now three blocks: (-∞,0) on shard0000, [0,5562] on shard0000 and [5562,+∞] on shard0001.

The data has been automatically assigned to these two pieces.

Note: The Sh.status () can be used to visually view the current fragmentation of the entire cluster, similar to the following:

mongos
> Sh.status (); 
---sharding Status---sharding version: {"_id": 1, "Version": 4, "Mincompatibleversion": 4, "CurrentVersion": 5, "Clusterid": ObjectId ("55a89388fe175819f8524cc2")} shards: {"_id": "shard0000", "host": "Localhost:1002"} {"_ ID ":" shard0001 "," host ":" Localhost:1003 "} databases: {" _id ":" admin "," partitioned ": false," PRIMARY ":" Con 	Fig "} {" _id ":" DBA "," partitioned ": true," PRIMARY ":" shard0000 "} dba.b Shard key: {" id ": 1} chunks:shard0000 2 shard0001 1 {"id": {"$minKey": 1}}-->> {"id": 0} on:shard0000 Timestamp (2, 1) {"id": 0}-->> ; {"id": 5562} on:shard0000 Timestamp (1, 3) {"id": 5562}-->> {"id": {"$maxKey": 1}} on:shard0001 Tim Estamp (2, 0) 

View the next data on the Shard server

--Shard 1
[root@pc config]# MONGO localhost:1002/admin
MongoDB shell version:2.6.9
connecting To:localhost : 1002/admin
> Show dbs;
Admin  (empty)
dba    0.078GB
local  0.078GB
> Use DBA;
Switched to DB DBA
> Db.b.find (). Count ();
5562
 
--Shard 2
[root@pc ~]# MONGO localhost:1003
MongoDB shell version:2.6.9
connecting To:localhost : 1003/test
> Show dbs;
Admin  (empty)
dba    0.078GB
local  0.078GB
> Use DBA;
Switched to DB DBA
> show tables;
b
system.indexes
> Db.b.find (). Count ();
94438

--This article references from: Building MongoDB Shard, MongoDB Shard Combat (ii): Sharding,mongodb authoritative guide.