Comparison of write time consumption between MongoDB3.0.6 wiredtiger and MMAPv1 Engines

Comparison of write time consumption between MongoDB3.0.6 wiredtiger and MMAPv1 Engines
I. Business Requirements: the database of an APP of the company has vertically separated logs from the business, splitting the original RAC into two sets. Currently, the database is relatively stable, the server load is also in the normal range, but the number of existing users is 4.5 million, and the number of daily active users reaches 1 million. 10 million logs are generated every day, with 18 million GB of data, while the number of target users is close, it is estimated that the number of databases per day will reach 60 million, and the requirement for querying a single record will be met. We plan to use MongoDB to replace Oracle RAC. Now we test the write comparison between MongoDB WiredTiger engine and MMAPv1 engine. Ii. Features of MongoDB3.0.x (compared with MongoDB2.6 and below ):

• Added the wiredtiger engine:

• Open-source storage engine;
• Supports multi-core CPU and full use of memory/chip-level cache (Note: In version 3.0.7, just released in December October 14, the memory operation performance is further improved );
• Based on B-TREE and LSM algorithm;
• Provides document-level concurrency control, similar to row-level locks of relational databases;
• Supports file compression (the snappy compression mechanism can consume an additional CPU by 5%, reduce the space usage by 70%, or adjust the compression ratio as needed). There are three compression types:

• Do not compress;
• Snappy compression: the default compression method. Snappy is a compression library used by many projects in Google's internal production environment, including BigTable, MapReduce, and RPC. The compression speed is faster than that of Zlib, however, the size of the compressed file is 20%-100% higher than that of Zlib. Snappy has a compression ratio of 1.5-1.7 for plain text and 2-4 for HTML, the compression ratio of JPEG, PNG, and other compressed data is 1.0. In I7 i7 5500u Single Core CPU test, the compression performance can be in 200 M/s-500M/s;
• Zlib compression: Z1ib is a free, universal, cross-platform, lossless data compression Development Library without any legal restrictions. Compared with Snappy compression, Zlib consumes high CPU performance and reduces compression speed, but the compression effect is good.

• MMAPv1 engine (the MMAP engine is used for MongoDB2.6 and later versions ):

• Collection-level concurrency control. In the MMAP version, only the database lock is provided (when a user operates on a collection, other collections are also suspended );
• Seamless migration (MMAP data can be migrated online to MMAPV1 or to wiredtiger engine ).

• Pluggable storage engine API (similar to mysql multi-engine driver)

• Use different data engines to meet different data needs;
• More scenarios will be expanded in the future.

• Ops manager (features provided by MongoDB Enterprise Edition)

• Web-based graphic Management Interface
• Reduce daily routine and configuration work

Iv. test environment:

Server hardware configuration: 172.16.16.169 CPU: 2 * E5620 @ 2.40 GHz, memory: 8 GB

Client hardware configuration: 172.16.40.92 CPU: 4 * I5-4300U @ 1.90 GHz, memory: 4G

Database: mongodb V3.0.6

Development Environment: python3.4.3 and pycharm4.5.4

Prerequisites: mongodb, python, and pycharm have been installed. test scenario 5: time consumption comparison for inserting 1 million pieces of data; 1. start MongoDB on the server side (the default is the MMAPv1 engine, the default port is 27017, start MongoDB in the MongoDB installation directory, and define the data and Log Path ):

#./mongod --dbpath=/data/db --logpath=/data/log 

2. log on to MongoDB:

#./mongo      

3. view the current engine status:

> db.serverStatus()  "storageEngine" : {                "name" : "mmapv1"

4. Start MongoDB on the server (WiredTiger engine, Port: 11111, start MongoDB in the MongoDB installation directory, define the data and Log Path, and define WiredTiger engine ):

#./mongod --dbpath=/data/wiredtiger --logpath=/data/wiredtiger/log  

5. log on to Mongodb that has started the WiredTiger engine:

#./mongo  127.0.0.1:11111 

6. view the current engine status:

> db.serverStatus()  "storageEngine" : {                "name" : "wiredTiger"

7. python connection to MongoDB MMAPv1 write Configuration:

import  timeimport datetimeimport  timeitimport  mathISOTIMEFORMAT = '%Y-%m-%d %X'from pymongo import MongoClientmc = MongoClient("172.16.16.169",27017)db = mc.usersdef dateDiffInSeconds(date1,date2):    timedelta = date2 - date1    return timedelta.days*24*3600 +timedelta.secondsdate1 = datetime.datetime.now()db.users.drop()for i  in range(0,1000000) :db.users.insert({"name":"ljai","age":i,"addr":"fuzhou"})c = db.users.find().count()print("count is ",c)date2 = datetime.datetime.now()print(date1)print(date2)print(dateDiffInSeconds(date1,date2),"seconds")mc.close()

8. Test results: It takes 12 minutes and 28 seconds to write 1 million pieces of data:

2. python connection to MongoDB WiredTiger write Configuration:

import  timeimport datetimeimport  timeitimport  mathISOTIMEFORMAT = '%Y-%m-%d %X'from pymongo import MongoClientmc = MongoClient("172.16.16.169",27017)db = mc.usersdef dateDiffInSeconds(date1,date2):    timedelta = date2 - date1    return timedelta.days*24*3600 +timedelta.secondsdate1 = datetime.datetime.now()db.users.drop()for i  in range(0,1000000) :db.users.insert({"name":"ljai","age":i,"addr":"fuzhou"})c = db.users.find().count()print("count is ",c)date2 = datetime.datetime.now()print(date1)print(date2)print(dateDiffInSeconds(date1,date2),"seconds")mc.close()

9. Test Result: it takes 10 minutes and 3 seconds to write 1 million pieces of data:

The WireTiger engine will write 1 million pieces of data faster than MMAPv1, and other performance tests will continue later.

For more information about MongoDB, see the following links:

MongoDB 3.0 official version released and downloaded

CentOS compilation and installation of MongoDB

CentOS compilation and installation of php extensions for MongoDB and mongoDB

CentOS 6 install MongoDB and server configuration using yum

Install MongoDB2.4.3 in Ubuntu 13.04

MongoDB beginners must read (both concepts and practices)

MongoDB Installation Guide for Ubunu 14.04

MongoDB authoritative Guide (The Definitive Guide) in English [PDF]

Nagios monitoring MongoDB sharded cluster service practice

Build MongoDB Service Based on CentOS 6.5 Operating System

MongoDB details: click here
MongoDB: click here

This article permanently updates the link address: