Remember that a failure is handled----the host unexpectedly shuts down after the MongoDB binary file is corrupted

Today, in a demo environment, our products experienced the entire engine room power outage, there is a MongoDB binary file corruption, the following is a breakdown of the analysis of the recording process:

1. In the customer support colleagues found that the entire engine room power outage and recovery, 3 MongoDB replication set, 1 hosts on the MongoDB service status error

2. Login background Discovery Replication set on each MongoDB host, the Mongod process is

3. On a well-served MongoDB host, log in to the database through MONGO, query the replication set status, find that the replication set status is normal, 1 primary+2 secondary, and optimedate time is consistent.

This time I was very curious, ordinarily MongoDB copy set state is normal, not to appear in 1 of the nodes on the Query MongoDB service status error situation.

Login error on the host, through the MONGO login database, at this time, very strange things come, the terminal directly error: "Bus error", Very strange ah, I this is the first time I met MONGO command reported this mistake. I feel like I'm having a weird incident. Then execute the MONGO-- version, the same error "Bus error."

This time, do not know how, suddenly remembered a very long time of a supernatural incident----the first brother to do products encountered such a problem: the same MongoDB RPM package, installed on a host of mongod binary files MD5 and expected to be different.

Then use md5sum to calculate this hint "Bus error" of the MONGO, the results of the terminal directly error "Input/output error", but use md5sum to calculate the same directory with the other several MongoDB related files did not error.

By this time, I realized what might have gone wrong with the operating system. Called the operating system group colleagues looked at the next,----just started to think it was only MONGO this binary file was modified by someone or other services, however, When we are ready to back up this corrupted MONGO binary file to another directory, the terminal continues to error "CP * * * * * input/output error".

At this point, the operating system group of colleagues determined: may be due to the computer room power outage, the host operating system corruption occurred in the file system.

In order to verify this speculation, next we restarted the server (fortunately not at this time), and then during the restart process, there was a hint that there was a file system corruption in the root partition and another partition. Follow the prompts to enter maintenance mode, after using fsck-y/dev/partition name repair, Again normal startup, the operating system will no longer error.

Finally, the MongoDB binary file error is processed by reloading the MongoDB RPM package. To this point, the MongoDB binary file corruption fix is complete.

I have always thought that the Linux file system is very stable, after this incident, found that the original is a misunderstanding, stability is only a relative concept.

Remember that a failure is handled----the host unexpectedly shuts down after the MongoDB binary file is corrupted