How to Increase MongoDBMapReduce speed by 20 times

Analysis has become an increasingly important topic in MongoDB because it is used in more and more large projects. People get tired of using different software for analysis (including Hadoop ),

Analysis has become an increasingly important topic in MongoDB because it is used in more and more large projects. People get tired of using different software for analysis (including Hadoop ),

Analysis has become an increasingly important topic in MongoDB because it is used in more and more large projects. People get tired of using different software for analysis (including Hadoop) and apparently need to transmit a large amount of overhead data.

MongoDB provides two built-in data analysis methods: Map Reduce and Aggregation. MR is flexible and easy to deploy. It works well through partitions and allows a large number of outputs. In MongoDB v2.4, MR replaces Spider Monkey with V8 by using the JavaScript engine, which improves performance a lot. The boss complained that it was too slow, especially compared with the runtime framework (using C ++. Let's see if we can squeeze some juice from it.

Exercise

Let's insert tens of millions of documents, each containing an integer from 0 to 1000000. This means that on average 10 documents will have the same value.

> For (var I = 0; I <10000000; ++ I) {db. uniques. insert ({dim0: Math. floor (Math. random () * 1000000 )});}
> Db. uniques. findOne ()
{"_ Id": ObjectId ("51d3c316acd412e22c188dec"), "dim0": 570859}
> Db. uniques. ensureIndex ({dim0: 1 })
> Db. uniques. stats ()
{
"Ns": "test. uniques ",
"Count": 10000000,
"Size": 360000052,
"AvgObjSize": 36.0000052,
"StorageSize": 582864896,
"NumExtents": 18,
"Nindexes": 2,
"LastExtentSize": 153874432,
"PaddingFactor": 1,
"SystemFlags": 1,
"UserFlags": 0,
"TotalIndexSize": 576040080,
"IndexSizes ":{
"_ Id _": 324456384,
"Dim0_1": 251583696
},
"OK": 1
}

Here, we want to calculate the number of different values. You can use the following MR tasks to easily complete this task:

> Db. runCommand (
{Mapreduce: "uniques ",
Map: function () {emit (this. dim0, 1 );},
Reduce: function (key, values) {return Array. sum (values );},
Out: "mrout "})
{
"Result": "mrout ",
"TimeMillis": 1161960,
"Counts ":{
"Inputs": 10000000,
"Emit": 10000000,
"Reduce": 1059138,
"Output": 999961
},
"OK": 1
}

As you can see in the output content, this takes about 1200 seconds (testing on the EC2 M3 instance ). There are 10 million maps and 1 million reduce, and 999961 documents are output. The result is as follows:

> Db. mrout. find ()
{"_ Id": 1, "value": 10}
{"_ Id": 2, "value": 5}
{"_ Id": 3, "value": 6}
{"_ Id": 4, "value": 10}
{"_ Id": 5, "value": 9}
{"_ Id": 6, "value": 12}
{"_ Id": 7, "value": 5}
{"_ Id": 8, "value": 16}
{"_ Id": 9, "value": 10}
{"_ Id": 10, "value": 13}
...

For more details, please continue to read the highlights on the next page:

MongoDB details: click here
MongoDB: click here

Recommended reading:

Java-based self-growth field in MongoDB

CentOS compilation and installation of MongoDB

CentOS compilation and installation of php extensions for MongoDB and mongoDB

CentOS 6 install MongoDB and server configuration using yum

Install MongoDB2.4.3 in Ubuntu 13.04

How to create a new database and set in MongoDB

MongoDB beginners must read (both concepts and practices)

MongoDB authoritative Guide (The Definitive Guide) in English [PDF]