Use aggregate to find duplicate data records in MongoDB

We know that MongoDB belongs to a document database, and its stored document types are JSON objects. Because of this feature, we often use MONGODB for data access in node. js. But because node. JS is executed asynchronously, this leads us to not guarantee that every database save operation is atomic. That is, if the client initiates the same event twice in a row, storing the data in the database is likely to cause the data to be saved repeatedly. In the case of high concurrency, even if you have already made very strict checks in your code, such as determining whether the data you want to save already exists before inserting the data, there is still a risk that the data will be stored repeatedly. Because in asynchronous execution, you have no way to guarantee which thread executes first, which thread executes, and all requests initiated by the client are not executed in the order we imagined. A good workaround is to create a unique index in all tables in the MONGO database. In fact, MongoDB defaults to creating a unique index of the _id field for all tables (which can be canceled). If you want to create an index from Mongoose.schema in node. js, you can refer to the following code:

varMongoose = require (' Mongoose '));varSchema =Mongoose. Schema;varCustomerschema =NewMongoose.         Schema ({cname:string, CellPhone, String, sender:string, tag:string, Behaviour:number, Createtime: { Type:date,default: Date.now}, current:{Type:boolean,default:true}}, {versionkey:false}); Customerschema.index ({cname:1,cellphone:1,sender:1,tag:1,behaviour:1}, {unique:true});

Module.exports = Mongoose.model (' customer ', customerschema);

In the model above, we define the structure of the table customer and create a unique index on the field cname,cellphone,sender,tag,behaviour with the index () method. This causes the database to throw an exception when duplicate data containing these fields is inserted. Borrowing Mongoose, if a database table has been created before and the program is running, when we modify the model and add an index, and then restart the app, as long as there is access to the model,Mongoose Automatically detects and creates an index. Of course, if the data is duplicated, index creation will fail. At this point we can let the database automatically delete duplicate data by adding the dropdups option when creating the index, such as:

true true});

However, according to MongoDB's official note, since the 3.0 version no longer use this option, but also does not provide alternative solutions. Looks like the official no longer provides the ability to automatically delete duplicate records when an index is created. So how can you quickly and efficiently find duplicate records and delete them? First we need to find these records and then delete them through the Remove () method. The following query statement can find records with duplicate data for a given field:

db.collection.aggregate ([  {$group: {     "$firstField", Secondfield: "$secondField" },     " $_id " },    1 }}   ,   {$match: {     1}}}   ])

Replace the value of the _id property to specify the field you want to judge. Accordingly, the code in node. JS is as follows:

var deferred = q.defer (); var group = {firstfield: "$firstField", Secondfield: "$secondField"};model.aggregate (). Group ({    _id: Group,    ' $_id '},    11}}). EXEC (Deferred.makenoderesolver ()); return deferred.promise;

The code above uses Q to replace callbacks in function execution. In the asynchronous programming of node. js, using Q to handle callbacks is a good choice.

The following is the result of the return:

/*1*/{    "Result" : [         {            "_ID" : {                "CellPhone": "15827577345",                "Actid": ObjectId ("5694565fa50fea7705f01789")            },            "UniqueIDs": [ObjectId ("569b5d03b3d206f709f97685"), ObjectId ("569b5d01b3d206f709f97684")            ],            "Count": 2.0000000000000000        },         {            "_ID" : {                "CellPhone": "18171282716",                "Actid": ObjectId ("566b0d8dc02f61ae18e68e48")            },            "UniqueIDs": [ObjectId ("566D16E6CF86D12D1ABCEE8B"), ObjectId ("566D16E6CF86D12D1ABCEE8A")            ],            "Count": 2.0000000000000000        }    ],    "OK": 1.0000000000000000}

As you can see from the results, there are two records of the same set of data, so the length of the result array returned is 2. The UniqueIDs property is an array that holds the value of the _id field of the repeating record, through which we can use the remove () method to find and delete the corresponding data.

Use aggregate to find duplicate data records in MongoDB