On the memory limit of MongoDB

I was also a recent contact with MongoDB, the understanding of MongoDB is not too deep, here to share a bit of personal experience in using MongoDB, the content may be wrong, for reference only.

MongoDB is a nosql non-relational database, is the big data with a relatively large number of non-relational database, providing high concurrency, which in the design of the disk when the data and memory mapping, then the need to read the disk, you can read the memory directly, improve speed, this is its advantage.

On the other hand, MongoDB does not directly manage memory, but uses the operating system's own memory relationship mechanism to manage it, so it is not possible to specify the maximum memory footprint directly in the configuration on MongoDB, and if you do not, the default MongoDB will set the system to 95% The memory is occupied (according to the time line on the system running state found), resulting in less memory available to the system, in this case, MongoDB may be inexplicably hang off (the log does not record the reason, there is no subsequent log records), the personal think may be the system available memory is low, When it comes to other situations where memory needs to be consumed, the system shoots MongoDB in order to release resources, but this is no evidence of its own speculation.

In order to control MongoDB this memory swallowed small monsters, tried to say some online methods, most of which use the Ulimit command to limit the use of MongoDB, but the effect is not good after the attempt.

Online says there are 2 kinds of restrictions on the parameters,

One, is to use ulimit-m XXX limit physical memory use, in the process to MongoDB Mongod configuration belongs to Mongod users, Mongod group, specifically set for Mongod user Ulimt-m 2097152 (in Kbytes, 2097152=2048*1024=2G), limiting the Mongod user's physical memory usage to 2G, the test found that this limitation did not work, and that the operation of 2G of physical memory would still grow.

Second, is to use ulimit-v XXX limit virtual memory use, this setting method is the same as above, set Ulimit-v 2097152, this will limit the Mongod user's memory usage is below 2G, but Mongod process after using virtual memory reaches 2G, If the data is also inserted into the MongoDB operation, the Mongod process can no longer get memory, the direct hanging off.

It seems that the method of using Ulimit to restrict MongoDB doesn't work, at least I'm not successful here.

Later on-line to find the next, found Cgroup this thing can limit memory use, so went to find the relevant information, and their own configuration test under, found can, now online system with this run for a week, temporarily did not find what problem, memory is limited.

Cgroup is a function of the Linux kernel to manage various resources of the system, including CPU, memory and other resources, the individual only looked at the use of memory. You can use the relevant command configuration for Cgroup directly, or you can manage it using a configuration file that is preconfigured with the Cgconfig service.

#########################################################################################

Above is nonsense, can not look, configuration are under!!!

#########################################################################################

Cgroup Limit Mognodb Memory usage

Here are the steps to proceed:

The operating system used is as follows, centos6.5x64

[[email protected] ~]# cat /etc/redhat-release 
CentOS release 6.5 (Final)
[[email protected] ~]# uname -a
Linux mongo1 2.6.32-431.23.3.el6.x86_64 #1 SMP Thu Jul 31 17:20:51 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

First, Installation:

A, Cgroup is a kernel component of Linux system, only need to install control interface program

Yum Install-y Libcgroup

B, start the Cgroup management program

Service Cgconfig Start

See if Cgconfig is started

Service Cgconfig Status

Set boot up

Chkconfig Cgconfig on

Second, the configuration:

A, look at the cgconfig configuration file first:

[[email protected] mongodb]# cat /etc/cgconfig.conf 
			mount {
				cpuset	= /cgroup/cpuset;
				cpu	= /cgroup/cpu;
				cpuacct	= /cgroup/cpuacct;
				memory	= /cgroup/memory;
				devices	= /cgroup/devices;
				freezer	= /cgroup/freezer;
				net_cls	= /cgroup/net_cls;
				blkio	= /cgroup/blkio;
			}

			group DBLimitedGroup {
					memory {
							memory.limit_in_bytes = "32212254720" ;

					}


			}

Above configuration file, mount{...} This section is the default in the/etc/cgconfig.conf generated after installing Libcgroup, with the purpose of defining the mount points of each subsystem, which can be used without a tube.

To limit MongoDB, a group name called Dblimitedgroup is defined, which defines the limit for memory memories, the limit is memory.limit_in_bytes, It has a value of 32212254720 (32212254720/1024/1024/1024=30GB) and the online server is 32G of memory, which is set to 30G to MongoDB. Depending on your situation, the value is in byte bytes, such as 1g=1*1024*1024*1024=1073741824 (byte)

b, then look at cgrules.conf.

cat  /etc/cgrules.conf 

# /etc/cgrules.conf
#The format of this file is described in cgrules.conf(5)
#manual page.
#
# Example:
#<user>		<controllers>	<destination>
#@student	cpu,memory	usergroup/student/
#peter		cpu		test1/
#%		memory		test2/
# End of file

*:mongod	memory		DBLimitedGroup/

This cgrules.conf is the default configuration for configuring the Cgexec command, followed by the cgexec to execute the Mongod program, "#" begins with a comment, without the tube, looking at the last line.

The above meaning is:

* All users and user groups

Mongod Mongod Program name (process)

Memory using the configuration in the memory subsystem

Dblimitedgroup a custom group in cgroup.conf

All together that is, when executing cgexec, if "Any user group executes the Mongod program, the executed process (and the mongod process) will satisfy all the limitations in Dblimitedgroup in the memory subsystem"

The limit in Dblimitedgroup is to limit memory usage to 32G.

This use of Cgroup limits the (physical) memory usage of the specified process (this is the mongod process).

Restart Cgconfig after configuration is complete

Service Cgconfig Restart

Third, running MongoDB

First Host:

cgexec   -g memory:DBLimitedGroup /usr/bin/mongod -f /etc/mongodb/shard11.conf
cgexec   -g memory:DBLimitedGroup /usr/bin/mongod -f /etc/mongodb/shard21.conf
cgexec   -g memory:DBLimitedGroup /usr/bin/mongod -f /etc/mongodb/mongosvr.conf
/usr/bin/mongos -f /etc/mongodb/mongos.conf

Second Host:

cgexec   -g memory:DBLimitedGroup /usr/bin/mongod -f /etc/mongodb/shard12.conf
cgexec   -g memory:DBLimitedGroup /usr/bin/mongod -f /etc/mongodb/shard22.conf
cgexec   -g memory:DBLimitedGroup /usr/bin/mongod -f /etc/mongodb/mongosvr.conf
/usr/bin/mongos -f /etc/mongodb/mongos.conf

Ground three hosts:

cgexec   -g memory:DBLimitedGroup /usr/bin/mongod -f /etc/mongodb/arb1.conf
cgexec   -g memory:DBLimitedGroup /usr/bin/mongod -f /etc/mongodb/arb2.conf
cgexec   -g memory:DBLimitedGroup /usr/bin/mongod -f /etc/mongodb/mongosvr.conf
/usr/bin/mongos -f /etc/mongodb/mongos.conf

I'm using the MongoDB cluster here, version of 3.2.4

The following are:

First Host:

650) this.width=650; "Src=" Http://s1.51cto.com/wyfs02/M02/82/03/wKioL1dHu8yw7K2MAAC8Acz3vKQ731.png-wh_500x0-wm_3 -wmp_4-s_2032556951.png "title=" 111.png "alt=" Wkiol1dhu8yw7k2maac8acz3vkq731.png-wh_50 "/>

Second Host:

650) this.width=650; "Src=" Http://s4.51cto.com/wyfs02/M02/82/04/wKiom1dHusWhyqigAADJ17PZ0oA573.png-wh_500x0-wm_3 -wmp_4-s_951635964.png "title=" 222.png "alt=" Wkiom1dhuswhyqigaadj17pz0oa573.png-wh_50 "/>

Third Host:

650) this.width=650; "Src=" Http://s2.51cto.com/wyfs02/M00/82/04/wKiom1dHuqvRQ0x2AACt65EjyNk640.png-wh_500x0-wm_3 -wmp_4-s_18284607.png "title=" 333.png "alt=" Wkiom1dhuqvrq0x2aact65ejynk640.png-wh_50 "/>

Because this article is about Cgroup limiting MONGO memory, the specific configuration of MongoDB is not listed.

This article is from the "Xiaotong blog" blog, make sure to keep this source http://xiaotong.blog.51cto.com/4312502/1783768

On the memory limit of MongoDB