Mongodb Sharding 分區

標籤：mongodb   sharding   分區   

Mongodb Sharding

  

介紹 Sharding in MongoDB 分區組建：

Shards(建議用Replica)、Query Routers、Config Servers。

 

Shards store the data. To provide high availability and dataconsistency, in a production sharded cluster, each shard is a replica set [1].For more information on replica sets, see ReplicaSets.

Query Routers, or mongos instances, interface with clientapplications and direct operations to the appropriate shard or shards. Thequery router processes and targets operations to shards and then returnsresults to the clients. A sharded cluster can contain more than one queryrouter to divide the client request load. A client sends requests to one queryrouter. Most sharded clusters have many query routers.

Config servers store thecluster’s metadata. This data contains a mapping of the cluster’s data set tothe shards. The query router uses this metadata to target operations tospecific shards. Production sharded clusters have exactly 3 config servers.

[1]	For development and testing purposes only, each shard can be a single mongod instead of a replica set. Donot deploy production clusters without 3 config servers.

  

分區類型:Range和Hash 

比較總結:線性用range散列用hash。

詳細的就不從官網複製了。

環境準備:分區結構分布如下：

 ShardServer 1：27020

ShardServer 2：27021

ShardServer 3：27022

ShardServer 4：27023

ConfigServer ：27100

RouteProcess：40000

ShardServer 5：27024(類比新增服務節點)

 

步驟一：啟動Shard Server 

mongod--port 27020 --dbpath=F:\DingSai\Mongodb\shard\rs1\data --logpath=F:\DingSai\Mongodb\shard\rs1\logs\mongodb.log --logappend

 mongod--port 27021 --dbpath=F:\DingSai\Mongodb\shard\rs2\data --logpath=F:\DingSai\Mongodb\shard\rs2\logs\mongodb.log --logappend 

mongod--port 27022 --dbpath=F:\DingSai\Mongodb\shard\rs3\data --logpath=F:\DingSai\Mongodb\shard\rs3\logs\mongodb.log --logappend

mongod--port 27023 --dbpath=F:\DingSai\Mongodb\shard\rs4\data  --logpath=F:\DingSai\Mongodb\shard\rs4\logs\mongodb.log--logappend

 

步驟二： 啟動ConfigServer 

mongod--port 27100 --dbpath=F:\DingSai\Mongodb\shard\config\data--logpath=F:\DingSai\Mongodb\shard\config\logs\mongodb.log --logappend

 注意：這裡我們完全可以像啟動普通mongodb服務一樣啟動，不需要添加—shardsvr和configsvr參數。因為這兩個參數的作用就是改變啟動連接埠的，所以我們自行指定了連接埠就可以。

 

步驟三： 啟動RouteProcess

mongos --port 40000 --configdblocalhost:27100 --logpath=F:\DingSai\Mongodb\shard\RouteProcess\logs\route.log --chunkSize 500

mongos啟動參數中，chunkSize這一項是用來指定chunk的大小的，單位是MB，預設大小為200MB.

 

步驟四： 配置Sharding

接下來，我們使用MongoDB Shell登入到mongos，添加Shard節點

bin/mongoadmin --port 40000

MongoDBshell version: 2.0.7

connectingto: 127.0.0.1:40000/admin

use admin

mongos>db.runCommand({ addshard:"localhost:27020" })

mongos>db.runCommand({ addshard:"localhost:27021" })

mongos>db.runCommand({ addshard:"localhost:27022" })

mongos>db.runCommand({ addshard:"localhost:27023" })

 

--可以添加節點名db.runCommand({addshard : "192.168.253.212:27017",  "name" : "XXX Server" });

 

--對資料庫 ding 啟用分區

mongos>db.runCommand({ enablesharding:"ding" })

 

 

--對資料庫  ding 下面的  c2  表，啟用分區--按照 _id 列進行分區

mongos>db.runCommand({shardcollection: "ding.c2", key: {_id:1}}

  

步驟五：測試插入資料:

mongoadmin --port 40000

--切換資料庫

useding

##插入5萬行

mongos>for(var i=0;i<500;i++){ db.c2.insert({name:‘dingsai‘+i,seq:i}) }

##查看記錄

mongos>  db.c2.find().count()

 

--查看分區狀態

 db.printShardingStatus()

或者sh.status();

 mongos> db.printShardingStatus()

---Sharding Status ---

  sharding version: {

        "_id" : 1,

        "version" : 4,

        "minCompatibleVersion" : 4,

        "currentVersion" : 5,

        "clusterId" :ObjectId("54dffdc6d33e0feb326a8f90")

}

  shards:

        { "_id" : "shard0000", "host" : "localhost:27020" }

        { "_id" : "shard0001", "host" : "localhost:27021" }

        { "_id" : "shard0002", "host" : "localhost:27022" }

        { "_id" : "shard0003", "host" : "localhost:27023" }

  databases:

         { "_id" : "ding", "partitioned" : true, "primary" : "shard0001" }

             ding.c2

                        shard key: {"_id" : 1 }

                        chunks:

                                shard0000       1

                                shard0001       1

                                shard0002       1

                        { "_id" : {"$minKey" : 1 } } -->> { "_id" :ObjectId("54e009d6752aea9c8fc3d25a") } on :shard0000 Timestamp(2, 0)

                        { "_id" :ObjectId("54e009d6752aea9c8fc3d25a") } -->> { "_id" :ObjectId("54e00a76752aea9c8fc3e684") } on :shard0001 Timestamp(3, 1)

                        { "_id" :ObjectId("54e00a76752aea9c8fc3e684") } -->> { "_id" :{ "$maxKey" : 1 } } on :shard0002 Timestamp(3, 0)

 

}

 

資料存放區在三個節點上

新增分區節點開啟服務:

bin/mongod--port 27024 --dbpath=F:\DingSai\Mongodb\shard\rs5\data --logpath=F:\DingSai\Mongodb\shard\rs5\logs\mongodb.log --logappend

登入RouteProcess:

mongoadmin --port 40000

useadmin

 

增加27024

mongos>db.runCommand({addshard : "localhost:27024"});

刪除分區節點

刪除sharding節點時,需要一定的時間，多次執行刪除語句可查看當前刪除中的狀態

 

刪除27021-shard0001分區開始刪除

mongos>  db.runCommand({removeShard :"shard0001" });

{

        "msg" : "drainingstarted successfully",

        "state" : "started",

        "shard" :"shard0001",

        "note" : "you need todrop or movePrimary these databases",

        "dbsToMove" : [

                "ding"

        ],

        "ok" : 1

}

 

 查看當前刪除狀態 刪除中

 

mongos>db.runCommand({removeShard : "shard0001" });

{

        "msg" : "draining ongoing",

        "state" : "ongoing",

        "remaining" : {

                "chunks" :NumberLong(0),

                "dbs" : NumberLong(1)

        },

        "note" : "you need todrop or movePrimary these databases",

        "dbsToMove" : [

                "ding"

        ],

        "ok" : 1

}

 

查看當前刪除狀態 刪除完成

mongos>db.runCommand({removeShard : "shard0001" });

{

        "ok" : 0,

        "errmsg" : "removeShardmay only be run against the admin database.",

        "code" : 13

}

 

再次查看parding狀態 

ongos>db.printShardingStatus()

--Sharding Status ---

 sharding version: {

       "_id" : 1,

       "version" : 4,

       "minCompatibleVersion" : 4,

       "currentVersion" : 5,

       "clusterId" :ObjectId("54dffdc6d33e0feb326a8f90")

 

 shards:

       { "_id" : "shard0000", "host" : "localhost:27020" }

       { "_id" : "shard0001", "host" : "localhost:27021",  "draining" :true }

       { "_id" : "shard0002", "host" : "localhost:27022" }

       { "_id" : "shard0003", "host" : "localhost:27023" }

       { "_id" : "shard0004", "host" : "localhost:27024" }

 databases:

        { "_id" : "ding", "partitioned" : true, "primary" : "shard0001" }

                ding.c2

                       shard key: {"_id" : 1 }

                       chunks:

                               shard0000       1

                               shard0003       1

                               shard0002       1

                       { "_id" : {"$minKey" : 1 } } -->> { "_id" :ObjectId("54e009d6752aea9c8fc3d25a") } on :shard0000 Timestamp(2, 0)

                       { "_id" :ObjectId("54e009d6752aea9c8fc3d25a") } -->> { "_id" :ObjectId("54e00a76752aea9c8fc3e684") } on :shard0003 Timestamp(4, 0)

                       { "_id" :ObjectId("54e00a76752aea9c8fc3e684") } -->> { "_id" :{ "$maxKey" : 1 } } on :shard0002 Timestamp(3, 0)

 

ongos>

 

發現：shard0001號機器狀態為draining:true

資料也已經從shard0001移到別的節點上。

 

 

 

節點損壞查詢:

1.  關閉27002連接埠節點 

2.查看損壞節點資訊：mongos>use config

switchedto db config

mongos>db.mongos.find()

{"_id" : "hpo-PC:40000", "ping" :ISODate("2015-02-15T03:36:40.670Z"), "up" : 5762,"waiting" : false, "mongoVersion" : "2.6.6" }

mongos>

 

3.重啟27002後正常使用 

步驟六：資料備份 

匯出ding資料庫 到ding檔案夾

mongodump-db ding -out F:\DingSai\Mongodb\bin\ding\ -h localhost:40000

 

總結：

什麼時候使用Sharding

1.  複製所有的寫入操作到主節點

2.  延遲的敏感性資料會在主節點查詢

3.  單個複本集限制在12個節點

4.  當請求量巨大時會出現記憶體不足。

5.  本地磁碟不足

6.  垂直擴充價格昂貴 

以上只是簡單的實現，官網推薦使用複本集(Replica Set)作為分區節點。解決系統單點問題。

改天有時間再實現。 

問題：

Windows32位有2G檔案限制，使用的時候注意。 

 

IMPORTANT

Ittakes time and resources to deploy sharding. If your system has already reachedor exceeded its capacity, it will be difficult to deploy sharding withoutimpacting your application.

 

Asa result, if you think you will need to partition your database in the future,do not wait until your system is over capacity to enable sharding. 

Mongodb Sharding 分區