MongoDB的mongos執行個體因無法分配mlock記憶體掛掉

標籤：分配   markdown   not   mon   nal   bug   god   har   cto   

問題版本

mongodb-v3.4.4

問題描述

mongos兩天死了兩次，死前遺言只有日誌：

2017-11-01T11:25:27.135+0800 F -        [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed to mlock: Cannot allocate memory2017-11-01T11:25:27.140+0800 I -        [NetworkInterfaceASIO-TaskExecutorPool-1-0] Fatal Assertion 28832 at src/mongo/base/secure_allocator.cpp 2462017-11-01T11:25:27.140+0800 I -        [NetworkInterfaceASIO-TaskExecutorPool-1-0]***aborting after fassert() failure2017-11-01T11:25:27.182+0800 I NETWORK  [thread2] connection accepted from 100.97.227.0:37132 #282914 (1156 connections now open)2017-11-01T11:25:27.191+0800 F -        [NetworkInterfaceASIO-TaskExecutorPool-1-0] Got signal: 6 (Aborted). 0x55db5fab1241 0x55db5fab0339 0x55db5fab081d 0x7f848d578390 0x7f848d1d3428 0x7f848d1d502a 0x55db5f2e90e7 0x55db5f3452cd 0x55db5f407733 0x55db5f4039ce 0x55db5f40538c 0x55db5f3da9c7 0x55db5f3fc16f 0x55db5f3fedaf 0x55db5f7dba02 0x55db5f7dc1be 0x55db5f7ddf64 0x55db5f7a69d4 0x55db5f7b9498 0x55db5f7b9ae2 0x55db5fb8edd4 0x55db5fb8efd1 0x55db5f7cf91f 0x55db5fe8e120 0x7f848d56e6ba 0x7f848d2a482d

關鍵點：Failed to mlock: Cannot allocate memory和Got signal: 6 (Aborted).

排查

mongodb的jira上有兩個issue跟這個一模一樣：

1. SERVER-29086
2. SERVER-28997

分析

如SERVER-28997

SaslSCRAMSHA1ClientConversations have a SCRAMSecrets which they ‘ll pull out of the cache. SCRAMSecrets allocate secure storage in their default constructor, so they may be populated. Instead, SaslSCRAMSHA1ClientConversation and the cache should store shared_ptrs to SCRAMSecret.

大致意思就是說SaslSCRAMSHA1ClientConversation這玩意兒每次初始化對象都會申請一片安全的儲存，這個會填充memlock地區，正確的做法應該是共用這片地區。

處理

按照SERVER-29086的意思，可以調整memlock大小，來緩解這個問題。

按照SERVER-28997的意思，這是3.4.4版本的bug，所以可以升級到bug-fixed版本3.4.6，這是最終的解決方案。

注意：要多多關注各種issue，jira，郵件清單。

MongoDB的mongos執行個體因無法分配mlock記憶體掛掉