MongoDB thread safe batch processing, mongodb thread Batch Processing

MongoDB thread safe batch processing, mongodb thread Batch Processing

Mongo batch processing tool:

Package com. saike. solr. server. util; import java.net. unknownHostException; import java. util. arrayList; import com. mongodb. basicDBObject; import com. mongodb. DB; import com. mongodb. DBCollection; import com. mongodb. DBCursor; import com. mongodb. DBObject; import com. mongodb. mongo; import com. mongodb. except exception; import com. mongodb. batch options;/*** batch processing tool class * @ author xieyong **/public class UtileMongDB {UtilThreadLocal <ArrayList <DBObject> localBatch; /** the mongo singleton object is thread-safe */private static Mongo mongo; private static DBCollection coll; // private static Log log = LogFactory. getLog (UtileMongDB. class); private static DB db; static {/** instantiate db */CATALOG options = new catalog options (); options. autoConnectRetry = true; options. connectionsperhost= 1000; options. maxWaitTime = 5000; options. socketTimeout = 0; options. connectTimeout = 15000; options. threadsAllowedToBlockForConnectionMultiplier = 5000; try {mongo = new Mongo (MongoDBConstant. performance_host, MongoDBConstant. cmd_port);} catch (UnknownHostException | unknown exception e) {e. printStackTrace ();} // boolean auth = db. authenticate (myUserName, myPassword);} public UtileMongDB () {try {localBatch = new UtilThreadLocal <ArrayList <DBObject> (ArrayList. class);} catch (Exception e) {e. printStackTrace () ;}}/*** return db object * @ return db */public static DB getDB () {if (db = null) {db = mongo. getDB (MongoDBConstant. MONGO_DB);} return db;}/*** returns mongo * @ return mongo connection pool */public static Mongo getMong () {return mongo ;} /*** read Collection * @ return mongo collection **/public static DBCollection getColl (String collname) {return getDB (). getCollection (collname);} public static DBCollection getColl () {return getDB (). getCollection (MongoDBConstant. MONGO_COLLECTION);}/** crud operation */public void addBatch (String key, String value) {BasicDBObject basicDB = new BasicDBObject (); basicDB. put (key, value);/** use the local variable of the thread here, and there is no need to have a race condition */localBatch. newGet (). add (basicDB);}/*** execute batch processing **/public void executeInsertBatch () {getColl (). insert (localBatch. get (); localBatch. get (). clear ();}/*** execute batch Delete */public void executeDeleteBatch () {ArrayList <DBObject> array = localBatch. get (); for (DBObject obj: array) {getColl (). remove (obj);} localBatch. get (). clear ();} public DBCursor query (String key, String value) {BasicDBObject basicDBObject = new BasicDBObject (); basicDBObject. put (key, value); return getColl (). find (basicDBObject );}}

 

 

ThreadLocal encapsulation:

Package com. saike. solr. server. util; import java. lang. reflect. constructor; /***** @ author xieyong ** @ param <T> the local thread variable object type */public class UtilThreadLocal <T> extends ThreadLocal <T> {/** parameter set */Object [] obj; /** instantiate the Constructor */Constructor <T> construct; /***** @ param clazz class of the local variable * @ param args constructor parameter * @ throws NoSuchMethodException * @ throws SecurityException */public UtilThreadLocal (Class clazz, Obje Ct... args) throws NoSuchMethodException, SecurityException {this. obj = obj; Class [] clazzs = null;/** new gets the parameter class for the constructor */if (args! = Null) if (args. length! = 0) {clazzs = new Class [args. length]; for (int I = 0; I <args. length; I ++) {clazzs [I] = args [I]. getClass () ;}} this. construct = clazz. getConstructor (clazzs);}/*** if no object exists in the current thread, create a new object * @ return */public T newGet () {T tar = super. get (); if (tar = null) {try {tar = construct. newInstance (obj); super. set (tar);} catch (Exception e) {e. printStackTrace () ;}} return tar ;}}

MongoDB batch data insertion failed

It's so sloppy...
// Your I equals 10, which is equivalent to the empty dbobjs set. Of course, no data can be saved .. For (int I = 10; I <10; I ++) {DBObject obj1 = new BasicDBObject (); obj1.put ("name", "I" + I); dbobjs. add (obj1);} is changed to the following. Try again. : For (int I = 0; I <10; I ++) {DBObject obj1 = new BasicDBObject (); obj1.put ("name", "I" + I); dbobjs. add (obj1 );}

How does MongoDB batch insert data?

This operation is not available because the underlying MongoDB does not support this operation.
From the oplog mode of MongoDB, we can see that each op of MongoDB is the smallest unit for independent execution. Therefore, there is no combination of multiple op into one execution. Therefore, this batch insert operation does not exist in various APIs.

In addition, according to my personal experience, my personal experience is correct:
This involves the essential differences between traditional relational databases and NoSQL, which are frequently used. Each NoSQL operation is lightweight and small, with no additional operations except Data Writing.
Analogy: MongoDB simply throws things into the corresponding Cabinet (database) when putting things (data), while MySQL needs to maintain communication with the deliverer (two-way connection ), storage of things in different la S (with mode ).
The batch insert of MySQL reduces the communication and sharding processes, and MongoDB itself does not have these processes. Therefore, MongoDB does not have the concept of batch insert.

Welcome to my blog for further discussion in Reference Materials
Reference: blog.sina.com.cn/u/1448370131