MongoDB Fragmentation (Single instance)

One: Fragment introduction

This is a database cluster system which expands the massive data level, the data table is stored on each node of sharding, the user can easily build a distributed MongoDB cluster by simple configuration.

MongoDB data blocks are called chunk. Each chunk is a continuous data record in the Collection, usually with a maximum size of 200MB, and a new block of data is generated when it is exceeded.

To build a MongoDB Sharding Cluster, you need three different roles:

Shard Server

That is, a fragment that stores the actual data, each shard can be a mongod instance, or it can be a set of replica set composed of Mongod instances. In order to implement each shard internal Auto-failover,mongodb, each shard is officially recommended as a set of replica set.

Config Server

In order to store a particular collection in multiple Shard, you need to specify a shard key for the collection, such as {age:1}, and the Shard key can determine which chunk the record belongs to. Config servers is used to store the configuration information for all shard nodes, the Shard key range for each chunk, chunk distribution in each Shard, collection configuration information for all DB and sharding in the cluster.

Route Process

This is a front-end routing, the client this access, and then ask config servers to which shard to query or save records, and then connect the corresponding shard to operate, and finally return the results to the client. The client simply sends the query or update request originally sent to the mongod to the routing Process without concern about which shard the record is stored on.

Here we build a simple Sharding Cluster on the same physical machine:

The architecture diagram is as follows:

Second: Experimental environment

Simulate a configuration server on the same machine, a mongos server, 2 slices (each fragment is a single instance)

Three: Experiment steps
3.1 Download Decompression MongoDB

It is supposed to download the decompression MongoDB on every server. Since all the simulations are done on the same machine, it is only necessary to extract it once.

TAR-XVF mongodb-linux-x86_64-2.6.9.tgz

# to facilitate management, move the installation files to/under Data

[ROOT@PC download]# MV Mongodb-linux-x86_64-2.6.9/data/mongodb

3.2 Creating a related directory

Create a location and log file for the MongoDB database

[root@pc mongodb]# cd/data/mongodb/
[root@pc mongodb]# mkdir data
[root@pc mongodb]# touch Logs
[root@pc mongodb]# ls-ltr Total
-rw-r--r--1 1046 1046  1359 Mar 22:49 README
-rw-r--r--1 1046 1046 34520 Ma R 22:49 gnu-agpl-3.0
-rw-r--r--1 1046 1046 17793 Mar 22:49 third-party-notices drwxr-xr-x
2 root root  4096 13:26 bin
drwxr-xr-x 2 root root 4096 June  13:34 data
-rw-r--r--1 root root     0 June 15 13 : Logs
 
#建配置信息服务器数据文件目录
[root@pc download]# mkdir-p/dbs/config
 
#建分片目录
mkdir-p/dbs/shard1
mkdir-p/dbs/shard2
 
#建日志目录
mkdir/logs
 
#建日志文件
touch/logs/config.log 
Touch Mongos.log Touch 
Shard1.log 

3.3 Configuration Path

[Root@pc dbs]# Vi/root/.bash_profile

Add at the end of the path line:

:/data/mongodb/bin

[Root@pc dbs]# Source/root/.bash_profile

3.4 Start Configuration server

The configuration server needs to be started first, and the MONGOs server needs to use the configuration information above. Configure the server to start just like a normal mongod.

Configuring a server does not require a lot of space and resources (200MB of actual data is about 1KB of configuration space)

[root@pc dbs]# mongod--dbpath=/dbs/config--logpath=/logs/config.log--fork--port 1000

About to fork the child process, the waiting until server is ready for connections.

Forked process:5055

Child process started successfully, parent exiting

[Root@pc ~]# Ps-ef | grep MONGO

Root 9691 1 3 22:31? 00:00:00 mongod--dbpath=/dbs/config--logpath=/logs/config.log--fork--port 1000

3.5 start MONGOs Server

[root@pc ~]# mongos--port 1001--logpath=/logs/mongos.log--configdb=192.168.6.51:1000--fork 2015-07-16T22
: 32:36.776-0700 warning:running with 1 config server should to do only for testing purposes and isn't recommended for Production about
to fork the child process, waiting until the server is ready for connections.
Forked process:9716 Child
process started successfully, parent exiting

--CONFIGDB here is the IP and port number where the configuration server resides.

When the--CONIGDB is configured, the IP address cannot be filled localhost or 127.0.0.1 or the following error message is returned when the fragment is added:

1.    {
2.            " OK ": 0,
3.            " ErrMsg ":" Can ' t use localhost as a shard since all shards need to communicate. Either use all shards and configdbs in localhost or all in actual IPs  host:192.168.6.51:1000 islocalhost:0 "
4.
  }

3.6 Start Fragmentation

#分片1

[root@pc ~]# mongod--dbpath=/dbs/shard1--logpath=/logs/shard1.log--fork--port 1002

About to fork the child process, the waiting until server is ready for connections.

Forked process:10583

Child process started successfully, parent exiting

#分片2

[root@pc ~]# mongod--dbpath=/dbs/shard2--logpath=/logs/shard2.log--fork--port 1003

About to fork the child process, the waiting until server is ready for connections.

Forked process:10659

Child process started successfully, parent exiting

3.7 Adding fragmentation

#注意: Adding fragmentation and shard data is done on the MONGOs server.

Log on to the MONGOs server,

[Root@pc mongodb]# MONGO Localhost:1001/admin

MongoDB Shell version:2.6.9

Connecting To:localhost:1001/admin

#添加分片

Mongos> Db.runcommand ({addshard: "localhost:1002", allowlocal:true});

{"shardadded": "shard0000", "OK": 1}

Mongos> Db.runcommand ({addshard: "localhost:1003", allowlocal:true});

{"shardadded": "shard0001", "OK": 1}

#查看所有的片

mongos> use config;
Switched to DB Config

Mongos> Db.shards.find ();

{"_id": "shard0000", "host": "Localhost:1002"}

{"_id": "shard0001", "host": "Localhost:1003"}

3.8 Segmentation Data

By default, each piece of data stored will not be fragmented, and the partitioning function needs to be turned on in both the database and the granularity of the collection.

Here take dba.b as an example

mongos> use DBA;

Switched to DB DBA

Mongos> db.createcollection ("B");

{"OK": 1}

fragmentation function of 3.8.1 Open library

mongos> use admin;

Switched to DB admin

Mongos> Db.runcommand ({"enablesharding": "DBA"});

{"OK": 1}

3.8.2 function of opening table

Mongos>  Db.runcommand ({"Shardcollection": "Dba.b", "key": {"id": 1}})
{"collectionsharded": "dba.b", "OK" : 1}
Note: You need to switch to the Admin library to execute the command.
key: The above key is called the piece Key (Shard key). MongoDB is not allowed to insert a document without a slice key. But to allow different types of pieces of the document, MongoDB internal to different types have a sort:
 
mongos> use config;
Switched to DB config
mongos> db.databases.find ();
{"_id": "admin", "partitioned": false, "PRIMARY": "Config"}
{"_id": "DBA", "partitioned": true, "PRIMARY": "shard0000"}
 
You can see that the DBA has a fragmented library.
 
mongos> db.chunks.find ();
{"_id": "Dba.b-id_minkey", "Lastmod": Timestamp (1, 0), "Lastmodepoch": ObjectId ("55a8c850fe175819f8525593"), "ns": "D Ba.b ", min": {"id": {"$minKey": 1}}, "Max": {"id": {"$maxKey": 1}}, "Shard": "shard0000"}
 
Chunks: Understanding M The key to Ongodb fragmentation mechanism is to understand chunks. MongoDB is not a fragment that stores an interval, but each fragment contains multiple intervals, each of which is a block.

3.9 Fragmentation Test

mongos
> Use DBA;
Switched to DB DBA mongos> for (i=0;i<100000;i++) {Db.b.insert ({"id": I, "Name": "Baidandan", "date": New Date ()});}
Writeresult ({"ninserted": 1}) mongos> use config;
Switched to DB config mongos> db.chunks.find (); {"_id": "Dba.b-id_minkey", "Lastmod": Timestamp (2, 1), "Lastmodepoch": ObjectId ("55a8c850fe175819f8525593"), "ns": "D Ba.b ", min": {"id": {"$minKey": 1}}, "Max": {"id": 0}, "Shard": "shard0000"} {"_id": "dba.b-id_0.0", "last  MoD ": Timestamp (1, 3)," Lastmodepoch ": ObjectId (" 55a8c850fe175819f8525593 ")," ns ":" Dba.b "," min ": {" id ": 0}," Max " : {"id": 5562}, "Shard": "shard0000"} {"_id": "dba.b-id_5562.0", "Lastmod": Timestamp (2, 0), "Lastmodepoch": OBJ Ectid ("55a8c850fe175819f8525593"), "ns": "Dba.b", "min": {"id": 5562}, "Max": {"id": {"$maxKey": 1}}, "Shard" : "Shard0001"} 

We can see that there are now three blocks: (-∞,0) on the shard0000, [0,5562] on the shard0000 and [5562,+∞] on the shard0001.

The data has been automatically assigned to these two slices.

Note: through Sh.status () you can visually view the current fragmentation of the entire cluster, similar to the following:

mongos
> Sh.status (); 
---Sharding Status---Sharding version: {"_id": 1, "Version": 4, "Mincompatibleversion": 4, "CurrentVersion": 5, "Clusterid": ObjectId ("55a89388fe175819f8524cc2")} shards: {"_id": "shard0000", "host": "Localhost:1002"} {"_ ID ":" shard0001 "," host ":" Localhost:1003 "} databases: {" _id ":" admin "," partitioned ": false," PRIMARY ":" Con 	Fig "} {" _id ":" DBA "," partitioned ": true," PRIMARY ":" shard0000 "} dba.b Shard key: {" id ": 1} chunks:shard0000 2 shard0001 1 {"id": {"$minKey": 1}}-->> {"id": 0} on:shard0000 Timestamp (2, 1) {"id": 0}-->> ; {"id": 5562} on:shard0000 Timestamp (1, 3) {"id": 5562}-->> {"id": {"$maxKey": 1}} on:shard0001 Tim Estamp (2, 0) 

Viewing data on a fragmented server

--Fragment 1
[root@pc config]# MONGO localhost:1002/admin
MongoDB shell version:2.6.9
connecting : 1002/admin
> Show dbs;
Admin  (empty)
dba    0.078GB
local  0.078GB
> Use DBA;
Switched to DB DBA
> Db.b.find (). Count ();
5562
 
--Fragment 2
[root@pc ~]# MONGO localhost:1003 MongoDB
shell version:2.6.9
connecting : 1003/test
> Show dbs;
Admin  (empty)
dba    0.078GB
local  0.078GB
> Use DBA;
Switched to DB DBA
> show tables;
b
system.indexes
> Db.b.find (). Count ();
94438

--This article reference from: Build MongoDB fragmentation, MongoDB fragment Combat (ii): Sharding,mongodb authoritative guide.