The MONGOs instance of MongoDB hangs out because mlock memory cannot be allocated

Issue version

mongodb-v3.4.4

Problem description

MONGOs died two times in two days, the last words of death only log:

2017-11-01T11:25:27.135+0800 F -        [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed to mlock: Cannot allocate memory
2017-11-01T11:25:27.140+0800 I -        [NetworkInterfaceASIO-TaskExecutorPool-1-0] Fatal Assertion 28832 at src/mongo/base/secure_allocator.cpp 246
2017-11-01T11:25:27.140+0800 I -        [NetworkInterfaceASIO-TaskExecutorPool-1-0]

***aborting after fassert() failure

2017-11-01T11:25:27.182+0800 I NETWORK  [thread2] connection accepted from 100.97.227.0:37132 #282914 (1156 connections now open)
2017-11-01T11:25:27.191+0800 F -        [NetworkInterfaceASIO-TaskExecutorPool-1-0] Got signal: 6 (Aborted).

 0x55db5fab1241 0x55db5fab0339 0x55db5fab081d 0x7f848d578390 0x7f848d1d3428 0x7f848d1d502a 0x55db5f2e90e7 0x55db5f3452cd 0x55db5f407733 0x55db5f4039ce 0x55db5f40538c 0x55db5f3da9c7 0x55db5f3fc16f 0x55db5f3fedaf 0x55db5f7dba02 0x55db5f7dc1be 0x55db5f7ddf64 0x55db5f7a69d4 0x55db5f7b9498 0x55db5f7b9ae2 0x55db5fb8edd4 0x55db5fb8efd1 0x55db5f7cf91f 0x55db5fe8e120 0x7f848d56e6ba 0x7f848d2a482d

Key points:Failed to mlock: Cannot allocate memoryandGot signal: 6 (Aborted).

Investigation

There are two issue on the Jira of MongoDB that are identical to this one:

1. SERVER-29086
2. SERVER-28997

Analysis

such as SERVER-28997

Saslscramsha1clientconversations has a scramsecrets which they ' ll pull out of the cache. Scramsecrets allocate secure storage in their default constructor, so they could be populated. Instead, saslscramsha1clientconversation and the cache should store shared_ptrs to Scramsecret.

Basically,SaslSCRAMSHA1ClientConversationit means this thing. Each initialization object will request a secure storage, this will fill the Memlock area, the correct way should be to share this area.

Processing

According to SERVER-29086 's meaning, you can adjust the memlock size to alleviate this problem.

According to SERVER-28997, this is the 3.4.4 bug, so you can upgrade to bug-fixed version 3.4.6, which is the final workaround.

Note: To pay more attention to various Issue,jira, mailing lists.

MONGOs instance of MongoDB hangs out because mlock memory cannot be allocated