Mongodb secondary上面replSet error RS102 too stale to catch up

1 http://101.130.92.32:28018/_replSet  監控網站上報錯:replSet error RS102 too stale to catch up
Tue Aug 13 05:18:06 [rsSync] replSet See

http://www.mongodb.org/display/DOCS/Resyncing+a+Very+Stale+Replica+Set+Member

           05:18:06 [rsSync] replSet error RS102 too stale to catch up
           05:20:07 [rsSync] replSet syncing to: 101.130.92.323:27018
           05:20:07 [rsSync] replSet error RS102 too stale to catch up, at lbgst from 101.130.92.323:27018
           05:20:07 [rsSync] replSet our last optime : Aug  6 19:18:05 52014bed:2
           05:20:07 [rsSync] replSet oldest at 101.130.92.323:27018 : Aug 10 10:58:26 52061cd2:17
           05:20:07 [rsSync] replSet See

http://www.mongodb.org/display/DOCS/Resyncing+a+Very+Stale+Replica+Set+Member

           05:20:07 [rsSync] replSet error RS102 too stale to catch up
           05:22:08 [rsSync] replSet syncing to: 101.130.92.323:27018
           05:22:08 [rsSync] replSet error RS102 too stale to catch up, at lbgst from 101.130.92.323:27018
           05:22:08 [rsSync] replSet our last optime : Aug  6 19:18:05 52014bed:2
           05:22:08 [rsSync] replSet oldest at 101.130.92.323:27018 : Aug 10 10:59:17 52061d05:3e

 

2 察看狀態,發現是recovring狀態    
[bguser@346437-bgstore_db1 ~]$ /db/mongodb/bin/mongo --port 27018
MongoDB shell version: 2.0.1
connecting to: 127.0.0.1:27018/test
RECOVERING> db.printReplicationInfo();
configured oplog size:   90000MB
log length start to end: 553610secs (153.78hrs)
oplog first event time:  Wed Jul 31 2013 09:31:15 GMT+0000 (UTC)
oplog last event time:   Tue Aug 06 2013 19:18:05 GMT+0000 (UTC)
now:                     Tue Aug 13 2013 06:03:40 GMT+0000 (UTC)
RECOVERING> rs.status();
{
        "set" : "rpl",
        "date" : ISODate("2013-08-13T06:03:49Z"),
        "myState" : 3,
        "syncingTo" : "101.130.92.323:27018",
        "members" : [
                {
                        "_id" : 0,
                        "name" : "101.130.92.32:27017",
                        "hbglth" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 8433,
                        "optime" : {
                                "t" : 1376373827000,
                                "i" : 16
                        },
                        "optimeDate" : ISODate("2013-08-13T06:03:47Z"),
                        "lastHbgrtbbgt" : ISODate("2013-08-13T06:03:47Z"),
                        "pingMs" : 0
                },
                {
                        "_id" : 4,
                        "name" : "101.130.92.32:27019",
                        "hbglth" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 8435,
                        "optime" : {
                                "t" : 0,
                                "i" : 0
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHbgrtbbgt" : ISODate("2013-08-13T06:03:47Z"),
                        "pingMs" : 0
                },
                {
                        "_id" : 5,
                        "name" : "101.130.92.32:27018",
                        "hbglth" : 1,
                        "state" : 3,
                        "stateStr" : "RECOVERING",
                        "optime" : {
                                "t" : 1375816685000,
                                "i" : 2
                        },
                        "optimeDate" : ISODate("2013-08-06T19:18:05Z"),  -- 同步時間也不對,是8月6日的,今天以及國內13日了
                        "self" : true
                },
                {
                        "_id" : 6,
                        "name" : "101.130.92.323:27017",
                        "hbglth" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 8433,
                        "optime" : {
                                "t" : 1376373827000,
                                "i" : 17
                        },
                        "optimeDate" : ISODate("2013-08-13T06:03:47Z"),
                        "lastHbgrtbbgt" : ISODate("2013-08-13T06:03:47Z"),
                        "pingMs" : 0
                },
                {
                        "_id" : 7,
                        "name" : "101.130.92.323:27018",
                        "hbglth" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 8435,
                        "optime" : {
                                "t" : 1376373827000,
                                "i" : 17
                        },
                        "optimeDate" : ISODate("2013-08-13T06:03:47Z"),
                        "lastHbgrtbbgt" : ISODate("2013-08-13T06:03:47Z"),
                        "pingMs" : 0
                }
        ],
        "ok" : 1
}
RECOVERING>     

 

3 去官網上面察看資料,發現如下
http://docs.mongodb.org/manual/tutorial/resync-replica-set-member/#replica-set-resync-by-copying

•Restart the mongod with an empty data directory and let MongoDB’s normal initial syncing fbgture restore the data.
This is the more simple option, but may take longer to replace the data.

重新啟動 步驟如下:
    (1) stop the failed mongod 停止正在啟動並執行同步異常的mongodb服務
    (2) delete all data in the dbpath (including subdirectories) 刪除mongodb目錄下所有的資料檔案包括其他的子目錄
    (3) start it and it will automatically resynchronize itself from primary  啟動mongodb服務，服務會自動從primary上重新同步資料過來。
 

4 重新sync後，狀態一直是 RECOVERING沒有變成secondary，不過資料都已經同步過去了。
RECOVERING> rs.status();
{
        "set" : "rpl",
        "date" : ISODate("2013-08-13T07:58:41Z"),
        "myState" : 3,
        "syncingTo" : "101.130.92.323:27018",
        "members" : [
                {
                        "_id" : 0,
                        "name" : "101.130.92.322:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 5629,
                        "optime" : {
                                "t" : 1376380719000,
                                "i" : 84
                        },
                        "optimeDate" : ISODate("2013-08-13T07:58:39Z"),
                        "lastHeartbeat" : ISODate("2013-08-13T07:58:39Z"),
                        "pingMs" : 0
                },
                {
                        "_id" : 4,
                        "name" : "101.130.92.322:27019",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 5629,
                        "optime" : {
                                "t" : 0,
                                "i" : 0
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2013-08-13T07:58:39Z"),
                        "pingMs" : 0
                },
                {
                        "_id" : 5,
                        "name" : "101.130.92.322:27018",
                        "health" : 1,
                        "state" : 3,
                        "stateStr" : "RECOVERING",
                        "optime" : {
                                "t" : 1376380721000,
                                "i" : 31
                        },
                        "optimeDate" : ISODate("2013-08-13T07:58:41Z"),
                        "self" : true
                },
                {
                        "_id" : 6,
                        "name" : "101.130.92.323:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 5629,
                        "optime" : {
                                "t" : 1376380719000,
                                "i" : 66
                        },
                        "optimeDate" : ISODate("2013-08-13T07:58:39Z"),
                        "lastHeartbeat" : ISODate("2013-08-13T07:58:39Z"),
                        "pingMs" : 0
                },
                {
                        "_id" : 7,
                        "name" : "101.130.92.323:27018",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 5629,
                        "optime" : {
                                "t" : 1376380719000,
                                "i" : 66
                        },
                        "optimeDate" : ISODate("2013-08-13T07:58:39Z"),
                        "lastHeartbeat" : ISODate("2013-08-13T07:58:39Z"),
                        "pingMs" : 0
                }
        ],
        "ok" : 1
}
RECOVERING>  

還需要繼續觀察一段時間，看是否最終會從RECOVERING變成secondary。

 

觀察了1天之後,還是RECOVRING, 日誌裡面也沒有報任何異常資訊.只好嘗試著重新啟動了下

kill 12292

/db/mongodb/bin/mongod -f /etc/mongodb/27018.conf

 

結果status變成了secondary了.