MONGOs process appears oom

A fault description

Online game clothing using MongoDB cluster shards, game profiles send query requests through MONGOs instances. Develop colleague to reflect program connection MONGOs error

Failed to connect to:10.4.4.66:28018:send_package:error reading from socket:the sockets is closed

After that, you can connect again.

10.4.4.66 is the IP address of MONGOs

Second fault analysis

1. Check if the MONGOs process is working properly and use MONGO to log on to the MONGOs terminal. Everything is fine.

2. Check the log file of the MONGOs to see if there are any exceptions

SERVER restarted *****thu Apr 15:02:01.726 [Mongosmain] MongoS version 2.4.6 starting:pid=7553 port=28018 64-bit HOST=XXXXXXX1 (--help for usage)

You can see that just during this time MONGOs restarted, after MONGOs reboot, there is a series of work to do, such as connecting config server to obtain the backend shard information. So the program will report the error.

3. View Linux system logs

$ sudo grep mongos /var/log/messages apr 10 15:35:38 localhost sz [32066]: [xxxx] check_mongos.sh/zmodem: 211 bytes, 229 bpsapr 23  14:50:18 localhost sz[5794]: [xxxxx] mongos/zmodem: 297 bytes, 151  bpsapr 23 15:01:55 localhost kernel: [20387]   497 20387    694326   427932   0       0              0 mongosapr 23 15:01:55  localhost kernel: Out of memory: Kill process 20387  (MONGOs)  score 890 or sacrifice childapr 23 15:01:55 localhost kernel:  Killed process 20387, UID 497,  (MONGOs)  total-vm:2777304kb, anon-rss : 1711700kb, file-rss:28kB 

It can be seen that the process was killed by the kernel because of mongos memory overflow.

4. Check the MONGOs pull script log

To prevent the MONGOs process from automatically restarting after it has been hung, we have added a timed task to check that the MONGOs process exists every other minute and not start the MONGOs

MONGOs is isn't runningstarting mongod:about to fork child process, waiting until server is a ready for connections.forked PR  Ocess:7553all output going to:/data/app_data/mongodb/log/mongos.logchild process started successfully, parent exiting[ OK]

You can see that the MONGOs process was killed by Oom, and the script detected that MONGOs was down and restarted MONGOs

5. Study the causes of MONGOs in the presence of Oom

This article is from the Linux SA John blog, so be sure to keep this source http://john88wang.blog.51cto.com/2165294/1637695

MONGOs process appears oom