(RPM) Mongdb Performance Optimization Collection

First, the database maximum connection number problem
When you find a lot of "connection refused because too many open connections:819" information in the background log, it is generally related to the value of the maximum connection you did not set.
By default, in a Linux system, the default number of MONGODB connections is 819, you can adjust this value appropriately, but note that this value is not infinite, can be set to 20000, see the official description of MongoDB.

We can specify the maximum number of connections by adding the--maxconns 10000 parameter when the database starts
You can also modify the mongodb.conf configuration file, in which add a sentence Maxconns = 10000 Save the exit and then start MongoDB just fine.
Of course this problem is also related to ulimit restrictions, you can manually modify the ulimit-n to change the number of open file.
If you want the value of the open file to take effect permanently, add the following four lines in the/etc/security/limits.conf, as the number is modified according to the system situation.
* Soft Nofile 102400 (* for all users)
* Hard Nofile 102400
Root soft Nofile 102400 (for root users)
Root Hard Nofile 102400


Then add the/etc/pam.d/login in the
Session required/lib64/security/pam_limits.so
....
Reboot can be applied permanently.

Two, virtual memory limit problem
After MongoDB master-slave configuration, start the Times wrong "Error:mmap failed with out of memory". This is because MongoDB will create "Creating Replication Oplog of SIZE:944MB" when set to master from the relationship, this oplog log should be put in memory.
Workaround:
(1) Set the size of the Oplog, with the parameter--oplogsize to specify, do not create the default 944M
(2) Release the limit of virtual memory (virtual machine default setting 512M), edit/etc/profile file to add ulimit-v Unlimited, use source/etc/profile let the settings take effect.
And then execute the master-slave boot command, OK.
MongoDB compared to eat memory, can also limit the memory usage of MongoDB, operation as follows
VI mongodb.conf
Increased Ulimit-m 2560000 (approx. 2.5G RAM)


A few things to note:
1. MongoDB "Mmap failed with out of memory" error occurs on 32-bit operating systems because MongoDB does not allow database files (cumulative sum) to exceed 2G in 32-bit platforms, while 64-bit platforms do not have this limitation. If this error is also reported on 64-bit platforms, it is generally due to insufficient virtual memory. You can edit the/etc/profile file to join Ulimit-v Unlimited, and use Source/etc/profile to make the settings take effect or restart.


2. The size of the oplog is not much related to memory, oplogsize is equivalent to the binlog of the MySQL database, and the data copied from the library is read from the Oplog, the local library.
--oplopgsize, specify the slave synchronization, the maximum size of the update log save, the latest version of MongoDB If you do not specify the parameters of the default is 5% of the hard disk space, if the settings are too small, slave synchronization and the main library is far more than the size of oplog. There may be inconsistent data.


See official documentation for details:
Http://www.mongodb.org/display/DOCS/Replication+Oplog+Length

3, using MongoDB recommended high-performance SAS hard disk, the pursuit of performance can consider the use of RAID10 hard disk.

Iii. the problem of MongoDB occupying too much space


1, space pre-allocation: In order to avoid the formation of excessive hard disk fragmentation, MongoDB each time the space shortage will be requested to generate a large chunk of hard disk space, and the amount of applications from 64M, 128M, 256M, such as the exponential increment, until 2G for the maximum volume of a single file. As the amount of data increases, you can see in their data directory the entire volume of files that have been incrementally generated.
2, the space occupied by the field name: In order to keep the structure information within each record used for querying, MongoDB needs to store the key-value of each field in the form of Bson, if the value field is not large relative to the key field, such as storing the data of numeric type, The overhead of the data is the largest. One way to reduce space usage is to take the field names as short as possible, so that it takes up less space, but this requires a tradeoff between legibility and space occupancy. It is recommended that you make the field name index, with each field named in one byte, so you don't have to worry about how long the field name takes. However, this method of indexing requires that the index value be replaced with the original value each time the query results, and then sent to the client, this substitution is also very time-consuming.
3, delete the record does not free space: This is easy to understand, in order to avoid the record deleted after the large-scale movement of data, the original record space is not deleted, only marked "deleted" can be reused later.


You can periodically run db.repairdatabase () to tidy up the record-freeing space, but the process is slow.

--------------------------
Optimize system configuration to improve MongoDB performance


Many system configurations affect the performance of the MongoDB runtime, and by optimizing these configurations, you can effectively improve MongoDB performance. This article is optimized for Linux systems, and the actual operation is done under CentOS 6. The article mainly comes from MongoDB Official document, http://docs.mongodb.org/manual/administration/production-notes/, at the same time combined with some practical experience of the author.


1, recommended to use 2.6.36 or above version of the Linux kernel
The author uses CentOS 6.3, the default kernel version is only 2.6.32. This optimization was not performed due to the larger impact of the upgrade kernel.


2. Using the EXT4 or XFS file system
MongoDB pre-allocates data files, and these files are very large, so it's best to use the EXT4 or XFS file system. Using EXT4 requires a kernel version of at least 2.6.23, using XFS requires a kernel version of at least 2.6.25.


3. Close the atime on the disk partition where the data file resides
Linux under the use of file Atime software is not many, common only mutt. If you do not use mutt such a dependent file atime time software, it is recommended to close off Atime. To reduce the impact of turning off atime, you can only close the atime on the disk partition where the MongoDB data file resides.
The following are the steps:
1. Switch to root identity
Su
2. Back up the current Fstab file
Cp/etc/fstab/etc/fstab.bak
3. Modify the Mount properties of the partition where the data files are located in the Fstab file
Modify the fourth section of the row of the partition where the data file is located and append "Noatime,nodiratime" to the back. If it turns out to be defaults, replace it directly.
Modified before:/dev/mapper/volgroup-lv_data/data EXT4 defaults 1 2
Modified:/dev/mapper/volgroup-lv_data/data ext4 noatime,nodiratime 1 2
Remount the partition, Mount-o remount/data/.


4. Modify the system limit parameter file descriptor number and user process to more than 20000.
Ulimit-n 64000
Ulimit-u 32000
This modification is only valid for the current session period, in order to remain valid after the next logon or restart, add the following to the "/etc/security/limits.conf" file.
* Soft Nofile 64000
* Hard Nofile 64000
* Soft Nproc 32000
* Hard Nproc 32000
Also, if the "/etc/security/limits.d/90-nproc.conf" file exists in your system, comment out the following line in the file.
#* Soft Nproc 1024


5. Do not use Hugepages
Cat/proc/sys/vm/nr_hugepages
Check the current hugepages settings to make sure it is 0. If not, perform the following actions.
echo 0 >/proc/sys/vm/nr_hugepages
In order for the settings to be valid after the next restart, add the following to the "/etc/sysctl.conf" file.
# sysctl Vm.nr_hugepages
Vm.nr_hugepages = 0


6. Disable NUMA
NUMA can be disabled in the BIOS, which requires a system restart. It can also be disabled for MONGODB processes only, recommended in this way.
The following are the steps:
Use Numactl to start Mongod

Numactl--interleave=all/usr/bin/local/mongod
Make sure that/proc/sys/vm/zone_reclaim_mode is 0, otherwise perform "echo 0 >/proc/sys/vm/zone_reclaim_mode", which does not require a restart Mongod.

7, set a smaller disk readahead parameter value, 32KB or 16KB is more appropriate
View the current value.

Blockdev--GETRA/DEV/SDA
Set to 32KB in sector size, a sector of 512 bytes, and 64 for 32KB.


Blockdev--setra 64/DEV/SDA
8. Set clock synchronization to ensure system time accuracy
Time accuracy is especially important for shard clusters. Install the NTPD service and set it to boot as boot.
Yum Install NTP
Chkconfig ntpd on


9. Disk and Storage System
Turn on swap. With regard to swap space, at least 16GB is required at least 8GB,64GB~256GB when the memory needs to be 2GB,4GB~16GB at a minimum of 4GB.
Disk arrays using RAID10,RAID5 and RAID6 do not provide sufficient performance, RAID0 write performance is good but read performance is poor, should avoid use. Use SSD disks as much as possible on a cost-permitting.
Do not use remote file systems, such as NFS.

(RPM) Mongdb Performance Optimization Collection