Analysis of speed problems in inserting large amounts of data into MongoDB database

analysis of speed problems in inserting large amounts of data into MongoDB database

Requirements background: A timed task produces thousands or more JSON data, this data is not fully written to the database, the next time the task of the data has been generated, the resulting data congestion how to solve.

When you initially used Springboot to insert data into a MONGODB database, you used the Save method in Mongotemplate to complete the data store operation.

The specific code is implemented as follows:

Jsonarray is the data I get from my scheduled tasks.
for (int i = 0;i < Jsonarray.size (); i++) {
Mongotemplate.save (Jsonarray.get (i), "Stored library name");
}

The data stored in this way is too slow because it is a storage that is traversed after one and is too inefficient.

You can use the Mongocollection.insertmany () method, which can be used to insert data in bulk and is highly efficient

The specific implementation code is as follows:

First, create a MongoDB connection Database tool class.
This class uses a method that does not have a password to connect to the database, and if your database has a password, refer to the code at the end of the article. That code is the use of user name, password to connect the database method, the code from the Novice tutorial, pro-Test effective.
public class Mongodbjdbc {
static String mongo_ip = "Database IP address";
static Integer Mongo_port = database port number, default is 27017;
public static mongocollection getmongodatabase (String databasename,string collectionname) {
Mongocollection mongocollection = null;
try {
Mongoclient mongoclient = new Mongoclient (mongo_ip,mongo_port);
Mongodatabase mongodatabase = mongoclient.getdatabase (databaseName);
Mongocollection = Mongodatabase.getcollection (CollectionName);
}catch (Exception e) {
System.out.println (E.getclass (). GetName () + ":" +e.getmessage ());
}
return mongocollection;
}
}

After you get to the Mongocollection object, use its Insertmany method to insert a collection.
public void InsertData () {
Use the Remove method first, delete a data, and then do the insert operation, in order to realize the batch data update function.
Query query = new query (Criteria.where ("Busi_type").
Is (Transformtime.getstringdate ()));
Mongotemplate.remove (Query, "collection name in Database");

list<document> list =//Gets the method that needs to be inserted into the database.

Use the tool class to get the Mongocollection object to the specified database
Mongocollection mongocollection = mongodbjdbc.getmongodatabase ("Database name", "Database collection name");
Mongocollection.insertmany (list);
}

Example of how to connect a database when a MongoDB database has a user name and password

Import java.util.ArrayList;
Import java.util.List;
Import com.mongodb.MongoClient;
Import com.mongodb.MongoCredential;
Import com.mongodb.ServerAddress;
Import Com.mongodb.client.MongoDatabase;

public class Mongodbjdbc {
public static void Main (string[] args) {
try {
Connect to a MongoDB service if a remote connection can replace "localhost" with the IP address of the server
ServerAddress () Two parameters are server address and port, respectively
ServerAddress serveraddress = new ServerAddress ("localhost", 27017);
list<serveraddress> Addrs = new arraylist<serveraddress> ();
Addrs.add (serveraddress);

Mongocredential.createscramsha1credential () Three parameters are user name database name password
Mongocredential credential = mongocredential.createscramsha1credential ("username", "databaseName", "password". ToCharArray ());
List<mongocredential> credentials = new arraylist<mongocredential> ();
Credentials.add (credential);

Obtaining MongoDB connections through connection authentication
Mongoclient mongoclient = new Mongoclient (addrs,credentials);

Connecting to a database
Mongodatabase mongodatabase = mongoclient.getdatabase ("DatabaseName");
System.out.println ("Connect to Database successfully");
} catch (Exception e) {
System.err.println (E.getclass (). GetName () + ":" + e.getmessage ());
}
}
}