MongoDB Performance Optimization

MongoDB is a high-performance extensible document-based NoSQL database, and high performance also requires configuration in several key dimensions, including hardware, application patterns, pattern design, indexing, disk I/O, and so on. Storage EngineWiredtiger is the default storage engine after 3.0, fine-grained concurrency control and data compression provide higher performance and storage efficiency. 3.0 Previously, the default MMAPV1 also improved performance. Multiple clock storage engines can be combined in a MONGODB replication set, and each instance implements different application requirements. HardwareMongoDB was originally designed to build clusters with horizontal scaling, rather than a higher-priced hardware upgrade. The use of replication to ensure high availability, automatic sharding to make the data evenly distributed across the nodes of the server; In-memory Computing provides higher computational performance, which only Enterprise editions have Hardware configuration Requirements

• Ensure memory settings meet performance requirements: Ensure memory > Index capacity + High frequency access to data capacity, set by Storage.wiredTiger.engineConfig.cacheSizeGB or--WIREDTIGERCACHESIZEGB. View and evaluate by monitoring the db.serverstatus () command
• Write-heavy applications with Ssd:sata interface SSD provide strong sequential write performance, can improve the writing speed of MongoDB data and jounal log files, and the most powerful is the random read performance, which conforms to MongoDB data Access VI mode. Deployment uses RAID-10, which reduces many limitations and provides higher performance than raid-5/raid-6.
• The storage engine configuration data compression and i/o-intensive Worloads:wiredtiger storage engine supports local compression, enabling better use of multicore new processor resources.
• Provide a dedicated server for each MONGODB service: Install multiple MongoDB instances on a single server, the system calculates the appropriate cache for each instance, and assigns a default cache_size for each instance. Adjusted by the Storage.wiredTiger.engineConfig.cacheSizeGB parameter.
• Use multiple query routing: It is best to deploy MONGOs on the application server by using more than one mongos on different servers.
• Leverage multi-core: The Wiredtiger storage engine is multi-threaded and can take full advantage of CPU multi-core. The MMAPV1 storage engine does not require more CPU cores because of its concurrency model.
• Turn on the NTP time synchronization service
• Disabling Numa:mongodb running on NUMA systems can cause operational problems, disabling NUMA requires configuring Mermory interleave policy

Scripting/etc/init.d/pro-startmongo...# Disable Numasysctl-w vm.zone_reclaim_mode=0 startup must specify Numactl mode Numactl--interleave =all <path> <options>

• disabling Thp:transparent Huge Pages (THP) is a Linux memory management policy that consumes a lot of memory

Scripting/etc/init.d/pro-startmongodb, adding the previous Disable NUMA section

# Disable Transparent hugespages Case$1 inchstart)if[-d/sys/kernel/mm/transparent_hugepage]; ThenThp_path=/sys/kernel/mm/Transparent_hugepageelif[-d/sys/kernel/mm/redhat_transparent_hugepage]; ThenThp_path=/sys/kernel/mm/Redhat_transparent_hugepageElsereturn0    fi     Echo 'never'> ${thp_path}/enabledEcho 'never'> ${thp_path}/Defrag Re='^[0-1]+$'    if[[ $(Cat${thp_path}/khugepaged/defrag) =~$re]]  Then# RHEL7      Echo 0> ${thp_path}/khugepaged/DefragElse# RHEL6      Echo 'No'> ${thp_path}/khugepaged/Defragfiunset re unset thp_path # disable NUMA sysctl-WVm.zone_reclaim_mode=0    ;;Esac

Change the script permissions and join the self-boot sudo chmod 755/etc/init.d/pro-startmongosudo chkconfig--add Pro-startmongo Pass the following test to see if the Thpcat/sys/kernel is off /mm/transparent_hugepage/enabledcat/sys/kernel/mm/transparent_hugepage/defrag shows success always madvise [never]

• Set Ulimit: The default Ulimit setting is too small, here is the recommended setting

-F (File size): Unlimited-t (CPU Time): Unlimited-v (Virtual memory): Unlimited-n (open files): 64000-m (memory size): UNL Imited-u (processes/threads): 64000 I am a new/etc/security/limits.d/99-mongodb-nproc.conf file, add the following configuration:mongod          soft    fsize     unlimitedmongod          HARD&NBS P   fsize     unlimitedmongod          soft    cpu    &NBSP ;  unlimitedmongod          hard    cpu       unlimitedmongod& nbsp         soft    as        unlimitedmongod        &NB Sp hard    as        unlimitedmongod          soft    Nofile     64000mongod          hard    nofile    64000mongod        &NBSP; soft    nproc     64000mongod          hard    nproc  &NB Sp  64000  application, pattern design

• Pattern Design: Correlation mode and nested mode, see, according to the application design the most reasonable pattern.

• The document key is named as short as possible, such as parkingid can be changed to PID. More than 16MB, large document use Gridfs as far as possible
• In a clustered environment, avoid scatter-gather queries. In fact, this depends on the sharding and sharding key choice, to try to meet the majority of business needs.
• Read/write separation design: In the allowable range of read delay, the separation of Read and write is a good choice, can greatly reduce the main node pressure.
• Similar relational database, some optimization suggestions: Only query and update the required fields, reduce bandwidth, avoid the use of negative criteria query, reasonable use of coverage index and consider the leftmost prefix matching principle;

hard disk I/O

• Pre-read settings: Use Blockdev--setra <value> commands to set the pre-read, the unit is 512B (sector size), not less than 32KB, also do not set too large, wasted I/O, set large pre-reading to facilitate sequential reading, the specific configuration according to business requirements set.
• Use the EXT4 or XFS file system. Do not use EXT3 oh.
• Disable access time settings. The operating system maintains the last access time of the file metadata, which means that every time the file system accesses a page, it commits a write operation, which reduces the performance of the entire database.

Edit the/etc/fstab file, specify Noatime and Nodiratime/dev/sdb/data ext4 defaults,noatime,nodiratime 1 2

• Disabling the Huges pages virtual memory page, MongoDB uses normal virtual memory pages for better performance, as mentioned earlier.
• With RAID10, performance is higher than RAID-5 or RAID-6
• Swap settings: Set sufficient swap space to prevent the Oom program to kill the Mongod process, for Linux under the MMAPV1 storage engine, do not use swap space, you can set a few swap space, Windows MMAPV1 storage engine needs to set swap space Wiredtiger storage engine, you need to set a large swap space. Using the Wiredtiger test process, found that for large queries, there is not enough swap space to even error. In/etc/sysctl.conf set vm.swappiness to 0, use memory as much as possible. Swap partitions as large as possible, here are some suggestions. During the test is 128G memory, I set the swap partition memory size is also 128G.

• Logs and data files are stored separately, logs are stored on an ordinary SAS disk, and data is stored on SSDs. Database reading is generally random reading, SSD performance is very good at random read, the log file is mainly sequential write, the speed itself is high, and SSD in sequential read and write performance is very bad, and more than normal SATA performance is lower. MongoDB logs have journal and Syslog,journal logs only in the journal directory below DBPath, and the system log can specify the log path. Journal can mount a SATA hard drive device. This enables log and data separation.
• Multipath settings: Set storage.wiredTiger.engineConfig.directoryForIndexes to separate indexes and data, set this parameter to true after setting up an index directory under DBPath to mount a hard disk device in the index directory so Indexes and data are separated. Set Directoryperdb to a different directory per database, preferably mounted on a different hard disk device for each directory.
• Compression: Wiredtiger provides two kinds of data compression methods for snappy and zlib. Snappy provides low compression ratio but low performance loss, zlib compression ratio, high performance loss, through Storage.wiredTiger.collectionConfig.blockCompressor parameter settings

MongoDB Performance Optimization