MongoDB Shard Cluster A sudden power outage caused some errors and the Shard could not be started (not properly closed)

Error:

1. /data/mongodb2/log/mongo.log

2015-09-14T05:41:49.288-0400 * * * * SERVER restarted * * * *

2015-09-14t05:41:49.293-0400 [Initandlisten] MongoDB starting:pid=13693 port=27017 dbpath=/data/mongodb2/db 64-bit Host=mongodb02.com

2015-09-14t05:41:49.293-0400 [Initandlisten]

2015-09-14t05:41:49.293-0400 [Initandlisten] * * Warning:you is running on a NUMA machine.

2015-09-14t05:41:49.293-0400 [Initandlisten] * * We suggest launching mongod like this to avoid performance proble Ms:

2015-09-14t05:41:49.293-0400 [Initandlisten] * * Numactl--interleave=all Mongod [Other options]

2015-09-14t05:41:49.565-0400 [Initandlisten]

2015-09-14t05:41:49.565-0400 [Initandlisten] DB version v2.6.4

2015-09-14t05:41:49.565-0400 [Initandlisten] git version:3a830be0eb92d772aa855ebb711ac91d658ee910

2015-09-14t05:41:49.565-0400 [Initandlisten] Build info:linux build7.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC x86_64 boost_lib_version=1_49

2015-09-14t05:41:49.565-0400 [Initandlisten] Allocator:tcmalloc

2015-09-14t05:41:49.565-0400 [Initandlisten] Options: {net: {http: {restinterfaceenabled:true, enabled:true}, MAXINC omingconnections:8000, port:27017}, Processmanagement: {fork:true}, replication: {replset: "Rep1"}, security: {Ke Yfile: "/data/mongodb2/keyfile/security"}, Storage: {dbPath: "/data/mongodb2/db"}, Systemlog: {destination: "File", Lo Gappend:true, Path: "/data/mongodb2/log/mongo.log"}}

2015-09-14t05:41:49.584-0400 [Initandlisten] Journal Dir=/data/mongodb2/db/journal

2015-09-14t05:41:49.585-0400 [Initandlisten] recover:no journal files present, no recovery needed

2015-09-14t05:41:49.914-0400 [Initandlisten] Preallocateisfaster=true 5.7

2015-09-14t05:41:50.258-0400 [Initandlisten] assertion:10334:bsonobj size:0 (0x0) is invalid. Size must be between 0 and 16793600 (16MB) First Element:eoo

2015-09-14t05:41:50.266-0400 [Initandlisten] Cache_common 0x11e6111 0x1187e49 0x116c9f6 0x116cf4c 0x77329b 0xefd3e8 0x8dd353 0x8b57e2 0x8c20d0 0x765f0e 0x76ab3f 0x76c62f 0x76cedb 0x76d475 0x76d699 0x7f14cf5ceaf5 0x764329

Mongod (_zn5mongo15printstacktraceerso+0x21) [0x11e6111]

Mongod (_zn5mongo10logcontextepkc+0x159) [0x1187e49]

Mongod (_zn5mongo11msgassertedeipkc+0xe6) [0x116c9f6]

Mongod () [0x116cf4c]

Mongod (_znk5mongo7bsonobj14_assertinvalidev+0x41b) [0x77329b]

Mongod (_znk5mongo7diskloc3objev+0x68) [0xefd3e8]

Mongod (_zn5mongo12indexcatalog4initev+0x213) [0x8dd353]

Mongod (_ZN5MONGO10COLLECTIONC1ERKNS_10STRINGDATAEPNS_16NAMESPACEDETAILSEPNS_8DATABASEE+0X2E2) [0x8b57e2]

Mongod (_zn5mongo8database13getcollectionerkns_10stringdatae+0x150) [0x8c20d0]

Mongod (_zn5mongo17checkforidindexesepns_8databasee+0x19e) [0x765f0e]

Mongod () [0x76ab3f]

Mongod (_ZN5MONGO14_INITANDLISTENEI+0X5DF) [0x76c62f]

Mongod (_ZN5MONGO13INITANDLISTENEI+0X1B) [0x76cedb]

Mongod () [0x76d475]

Mongod (main+0x9) [0x76d699]

/lib64/libc.so.6 (__LIBC_START_MAIN+0XF5) [0X7F14CF5CEAF5]

Mongod () [0x764329]

2015-09-14t05:41:50.267-0400 [Initandlisten] exception in initandlisten:10334 bsonobj size:0 (0x0) is invalid. Size must be between 0 and 16793600 (16MB) first Element:eoo, terminating

2015-09-14t05:41:50.267-0400 [Initandlisten] Dbexit:

2015-09-14t05:41:50.267-0400 [Initandlisten] shutdown:going to close listening sockets ...

2015-09-14t05:41:50.267-0400 [Initandlisten] shutdown:going to flush diaglog ...

2015-09-14t05:41:50.267-0400 [Initandlisten] shutdown:going to close sockets ...

2015-09-14t05:41:50.267-0400 [Initandlisten] shutdown:waiting for FS preallocator ...

2015-09-14t05:41:50.267-0400 [Initandlisten] shutdown:lock for final commit ...

2015-09-14t05:41:50.267-0400 [Initandlisten] shutdown:final commit ...

2015-09-14t05:41:50.267-0400 [Initandlisten] shutdown:closing All Files ...

2015-09-14t05:41:50.268-0400 [Initandlisten] Closeallfiles () finished

2015-09-14t05:41:50.268-0400 [Initandlisten] journalcleanup ...

2015-09-14t05:41:50.268-0400 [Initandlisten] Removejournalfiles

2015-09-14t05:41:50.269-0400 [Initandlisten] shutdown:removing FS lock ...

2015-09-14t05:41:50.269-0400 [Initandlisten] dbexit:really exiting now

Systemctl Status Mongodb.service

Mongodb.service-mongodb

Loaded:loaded (/usr/lib/systemd/system/mongodb.service; enabled)

Active:failed (Result:exit-code) since Mon 2015-09-14 05:25:06 EDT; 1h 10min ago

process:13499 Execstart=/usr/local/mongodb/bin/mongod--dbpath=/data/mongodb2/db--logpath=/data/mongodb2/log/ Mongo.log--fork--port 27017--logappend--replset rep1--rest--httpinterface--maxconns=8000--keyfile=/data/mongodb2 /keyfile/security (code=exited, status=100)

Sep 05:25:05 mongodb02.com systemd[1]: Starting MongoDB ...

Sep 05:25:05 mongodb02.com mongod[13499]: about-to-fork child process, waiting until server is a ready for connections.

Sep 05:25:05 mongodb02.com mongod[13499]: forked process:13501

Sep 05:25:06 mongodb02.com mongod[13499]: Error:child process failed, exited with ERROR number 100

Sep 05:25:06 mongodb02.com systemd[1]: Mongodb.service:control process exited, code=exited status=100

Sep 05:25:06 mongodb02.com systemd[1]: Failed to start MongoDB.

Sep 05:25:06 mongodb02.com systemd[1]: Unit Mongodb.service entered failed state.

[Email protected] ~]#/usr/local/mongodb/bin/mongod--dbpath=/data/mongoarb/arbiter--logpath=/data/mongoarb/log/ Arbiter.log--fork--port 27027--logappend--replset rep1--keyfile=/data/mongoarb/keyfile/security--rest

2015-09-14t06:57:48.025-0400 * * WARNING:--rest is specified without--httpinterface,

2015-09-14t06:57:48.025-0400 * * Enabling HTTP interface

About-to-fork child process, waiting until server was ready for connections.

Forked process:14495

Error:child process failed, exited with error number 1

Because the child process failed to prompt

[Email protected] ~]#/usr/local/mongodb/bin/mongod--dbpath=/data/mongodb2/db--logpath=/data/mongodb2/log/ Mongo.log--fork--port 27017--logappend--replset rep1--rest--httpinterface--maxconns=8000--keyfile=/data/mongodb2 /keyfile/security

About-to-fork child process, waiting until server was ready for connections.

Forked process:14545

Error:child process failed, exited with error number 100

Initialization, and also prompts the child process to fail to start.

MongoDB Shard Cluster A sudden power outage caused some errors and the Shard could not be started (not properly closed)