MongoDB Shard Cluster A sudden power outage caused some errors and the Shard could not be started

Today suddenly the power outage caused the MongoDB shard cluster to cause this error, temporarily do not know how to solve, if someone knows please reply me

And now put the record down and wait for the post-processing.

Fri 8 10:49:52.165 [Initandlisten] connection accepted from 172.16.0.115:59542 #2 (2 connections now OPEN)
Fri 8 10:49:52.954 [Initandlisten] connection accepted from 172.16.0.103:33501 #3 (2 connections now OPEN)
Fri 8 10:49:52.954 [Initandlisten] connection accepted from 172.16.0.103:33502 #4 (3 connections now OPEN)
Fri 8 10:49:52.955 [conn3] End Connection 172.16.0.103:33501 (2 connections now OPEN)
Fri 8 10:49:55.104 [Rsbackgroundsync] replset syncing to:172.16.0.115:11731
Fri 8 10:49:55.108 [Rssync] replset still syncing, not yet to Minvalid Optime 53e421d7:3a
Fri 8 10:49:55.154 [Rssync] GetFile (): n=808581868
Fri 8 10:49:55.154 [Rssync] Assertion:10295:getfile (): Bad file number value (corrupt db?): Run Repair
0xde0151 0xda188b 0x8d5465 0xb89952 0x818116 0x818244 0x81a4d4 0xa62bd5 0xa62c1c 0xab959c 0xa6c93c 0xc222ab 0xc28184 0xc28 728 0xc2b45d 0xc2b73c 0xe289d9 0x3dffe0683d 0x3dfeed503d
/home/mongodb/bin/mongod (_zn5mongo15printstacktraceerso+0x21) [0xde0151]
/home/mongodb/bin/mongod (_zn5mongo11msgassertedeipkc+0x9b) [0xda188b]
/home/mongodb/bin/mongod (_zn5mongo8database7getfileeiib+0x395) [0x8d5465]
/home/mongodb/bin/mongod (_znk5mongo7diskloc3recev+0x42) [0xb89952]
/home/mongodb/bin/mongod (_znk5mongo16namespacedetails11incapextenterkns_7diskloce+0x26) [0x818116]
/home/mongodb/bin/mongod (_ZN5MONGO16NAMESPACEDETAILS10__CAPALLOCEI+0XC4) [0x818244]
/home/mongodb/bin/mongod (_ZN5MONGO16NAMESPACEDETAILS11CAPPEDALLOCEPKCI+0X2E4) [0X81A4D4]
/home/mongodb/bin/mongod (_zn5mongo16namespacedetails6_allocepkci+0x25) [0xa62bd5]
/home/mongodb/bin/mongod (_ZN5MONGO16NAMESPACEDETAILS5ALLOCEPKCI+0X3C) [0xa62c1c]
/home/mongodb/bin/mongod (_ZN5MONGO11DATAFILEMGR17FAST_OPLOG_INSERTEPNS_16NAMESPACEDETAILSEPKCI+0X1EC) [0xab959c ]
/home/mongodb/bin/mongod (_zn5mongo11_logopobjrserkns_7bsonobje+0x4cc) [0xa6c93c]
/home/mongodb/bin/mongod (_ZN5MONGO7REPLSET8SYNCTAIL15APPLYOPSTOOPLOGEPST5DEQUEINS_7BSONOBJESAIS3_EE+0X4B) [ 0XC222AB]
/home/mongodb/bin/mongod (_zn5mongo7replset8synctail16oplogapplicationev+0x584) [0xc28184]
/home/mongodb/bin/mongod (_ZN5MONGO11REPLSETIMPL11_SYNCTHREADEV+0XB8) [0xc28728]
/home/mongodb/bin/mongod (_zn5mongo11replsetimpl10syncthreadev+0x2d) [0xc2b45d]
/home/mongodb/bin/mongod (_ZN5MONGO15STARTSYNCTHREADEV+0X6C) [0xc2b73c]
/home/mongodb/bin/mongod [0XE289D9]
/lib64/libpthread.so.0 [0x3dffe0683d]
/lib64/libc.so.6 (CLONE+0X6D) [0x3dfeed503d]
Fri 8 10:49:55.166 [Rssync] replset syncthread:10295 getFile (): Bad file number value (corrupt db?): Run Repair
Fri 8 10:49:56.105 [rssyncnotifier] replset setting Oplog notifier to 172.16.0.115:11731
Fri 8 10:50:02.191 [conn2] End Connection 172.16.0.115:59542 (1 connection now OPEN)
Fri 8 10:50:02.191 [Initandlisten] connection accepted from 172.16.0.115:59549 #5 (2 connections now OPEN)
Fri 8 10:50:06.228 [repl writer worker 1] assertion:10334:bsonobj size:1342177280 (0x00000050) is invalid. Size must be between 0 and 16793600 (16MB) First element:.:? type=113
0xde0151 0xda188b 0xda1dcc 0x6ec92f 0xa2aad8 0xa2bfcc 0xa2c370 0x7fbcf0 0x8016d0 0x80180f 0x80180f 0x80190c 0x814173 0x810 5f1 0x813291 0xb60b84 0xb6245e 0xb626be 0xa7c88b 0xa73af7
/home/mongodb/bin/mongod (_zn5mongo15printstacktraceerso+0x21) [0xde0151]
/home/mongodb/bin/mongod (_zn5mongo11msgassertedeipkc+0x9b) [0xda188b]
/home/mongodb/bin/mongod [0XDA1DCC]
/home/mongodb/bin/mongod (_ZNK5MONGO7BSONOBJ14_ASSERTINVALIDEV+0X5BF) [0x6ec92f]
/home/mongodb/bin/mongod (_znk5mongo5keyv16tobsonev+0x98) [0xa2aad8]
/home/mongodb/bin/mongod (_ZNK5MONGO5KEYV113COMPAREHYBRIDERKS0_RKNS_8ORDERINGE+0X3C) [0XA2BFCC]
/home/mongodb/bin/mongod (_znk5mongo5keyv19wocompareerks0_rkns_8orderinge+0x230) [0xa2c370]
/home/mongodb/bin/mongod (_ZNK5MONGO11BTREEBUCKETINS_12BTREEDATA_V1EE4FINDERKNS_12INDEXDETAILSERKNS_5KEYV1ERKNS _7DISKLOCERKNS_8ORDERINGERIB+0X120) [0x7fbcf0]
/home/mongodb/bin/mongod (_znk5mongo11btreebucketins_12btreedata_v1ee6locateerkns_12indexdetailserkns_ 7DISKLOCERKNS_5KEYV1ERKNS_8ORDERINGERIRBS8_I+0X60) [0x8016d0]
/home/mongodb/bin/mongod (_znk5mongo11btreebucketins_12btreedata_v1ee6locateerkns_12indexdetailserkns_ 7disklocerkns_5keyv1erkns_8orderingerirbs8_i+0x19f) [0x80180f]
/home/mongodb/bin/mongod (_znk5mongo11btreebucketins_12btreedata_v1ee6locateerkns_12indexdetailserkns_ 7disklocerkns_5keyv1erkns_8orderingerirbs8_i+0x19f) [0x80180f]
/home/mongodb/bin/mongod (_znk5mongo11btreebucketins_12btreedata_v1ee6locateerkns_12indexdetailserkns_ 7DISKLOCERKNS_7BSONOBJERKNS_8ORDERINGERIRBS8_I+0X8C) [0x80190c]
/home/mongodb/bin/mongod (_zn5mongo15btreecursorimplins_12btreedata_v1ee7_locateerkns_7bsonobjerkns_7diskloce+ 0X93) [0x814173]
/home/mongodb/bin/mongod (_ZN5MONGO11BTREECURSOR33INITWITHOUTINDEPENDENTFIELDRANGESEV+0X1F1) [0X8105F1]
/home/mongodb/bin/mongod (_zn5mongo11btreecursor4makeepns_16namespacedetailserkns_12indexdetailserkns_ 7BSONOBJES8_BI+0XA1) [0x813291]
/home/mongodb/bin/mongod (_ZNK5MONGO15CURSORGENERATOR14SHORTCUTCURSOREV+0X6C4) [0xb60b84]
/home/mongodb/bin/mongod (_zn5mongo15cursorgenerator8generateev+0x3e) [0xb6245e]
/home/mongodb/bin/mongod (_zn5mongo25namespacedetailstransient9getcursorerkns_10stringdataerkns_7bsonobjes6_ RKNS_24QUERYPLANSELECTIONPOLICYERKN5BOOST10SHARED_PTRIKNS_11PARSEDQUERYEEEBPNS_16QUERYPLANSUMMARYE+0X3E) [ 0XB626BE]
/home/mongodb/bin/mongod (_ZN5MONGO13DELETEOBJECTSEPKCNS_7BSONOBJEBBBPNS_11REMOVESAVERE+0X14B) [0xa7c88b]
/home/mongodb/bin/mongod (_ZN5MONGO21APPLYOPERATION_INLOCKERKNS_7BSONOBJEBB+0XC57) [0xa73af7]
Fri 8 10:50:06.241 [repl writer worker 1] error:writer worker caught Exception:bsonobj size:1342177280 (0x00000050 ) is invalid. Size must is between 0 and 16793600 (16MB) First element:.:? type=113 on: {Ts:timestamp 1407459783000|288, H:-344163686 8876691989, V:2, op: "D", NS: "mongotest1000. S270 ", Frommigrate:true, O: {_id:objectid (' 53dc305d31c6e0ce936d37ac ')}}
Fri 8 10:50:06.241 [repl writer worker 1] Fatal assertion 16360
0xde0151 0xd9ff43 0xc28aac 0xdada91 0xe289d9 0x3dffe0683d 0x3dfeed503d
/home/mongodb/bin/mongod (_zn5mongo15printstacktraceerso+0x21) [0xde0151]
/home/mongodb/bin/mongod (_ZN5MONGO13FASSERTFAILEDEI+0XA3) [0xd9ff43]
/home/mongodb/bin/mongod (_zn5mongo7replset14multisyncapplyerkst6vectorins_7bsonobjesais2_eepns0_8synctaile+ 0X12C) [0XC28AAC]
/home/mongodb/bin/mongod (_zn5mongo10threadpool6worker4loopev+0x281) [0xdada91]
/home/mongodb/bin/mongod [0XE289D9]
/lib64/libpthread.so.0 [0x3dffe0683d]
/lib64/libc.so.6 (CLONE+0X6D) [0x3dfeed503d]
Fri 8 10:50:06.245 [repl writer worker 1]

Aborting after Fassert () failure

Fri 8 10:50:06.245 Got signal:6 (aborted).

Fri 8 10:50:06.250 BackTrace:
0xde0151 0x6d0529 0x3dfee30330 0x3dfee302c5 0x3dfee31d70 0xd9ff7e 0xc28aac 0xdada91 0xe289d9 0x3dffe0683d 0x3dfeed503d
/home/mongodb/bin/mongod (_zn5mongo15printstacktraceerso+0x21) [0xde0151]
/home/mongodb/bin/mongod (_zn5mongo10abruptquitei+0x399) [0x6d0529]
/lib64/libc.so.6 [0x3dfee30330]
/lib64/libc.so.6 (GSIGNAL+0X35) [0X3DFEE302C5]
/lib64/libc.so.6 (abort+0x110) [0X3DFEE31D70]
/home/mongodb/bin/mongod (_ZN5MONGO13FASSERTFAILEDEI+0XDE) [0xd9ff7e]
/home/mongodb/bin/mongod (_zn5mongo7replset14multisyncapplyerkst6vectorins_7bsonobjesais2_eepns0_8synctaile+ 0X12C) [0XC28AAC]
/home/mongodb/bin/mongod (_zn5mongo10threadpool6worker4loopev+0x281) [0xdada91]
/home/mongodb/bin/mongod [0XE289D9]
/lib64/libpthread.so.0 [0x3dffe0683d]
/lib64/libc.so.6 (CLONE+0X6D) [0x3dfeed503d]

This article is from the "Fly Hung 膤" blog, be sure to keep this source http://jxzhfei.blog.51cto.com/1382161/1537319