MongoDB thread safe batch processing, mongodb thread Batch Processing

Source: Internet
Author: User
Tags log log solr

MongoDB thread safe batch processing, mongodb thread Batch Processing

Mongo batch processing tool:

Package com. saike. solr. server. util; import java.net. unknownHostException; import java. util. arrayList; import com. mongodb. basicDBObject; import com. mongodb. DB; import com. mongodb. DBCollection; import com. mongodb. DBCursor; import com. mongodb. DBObject; import com. mongodb. mongo; import com. mongodb. except exception; import com. mongodb. batch options;/*** batch processing tool class * @ author xieyong **/public class UtileMongDB {UtilThreadLocal <ArrayList <DBObject> localBatch; /** the mongo singleton object is thread-safe */private static Mongo mongo; private static DBCollection coll; // private static Log log = LogFactory. getLog (UtileMongDB. class); private static DB db; static {/** instantiate db */CATALOG options = new catalog options (); options. autoConnectRetry = true; options. connectionsperhost= 1000; options. maxWaitTime = 5000; options. socketTimeout = 0; options. connectTimeout = 15000; options. threadsAllowedToBlockForConnectionMultiplier = 5000; try {mongo = new Mongo (MongoDBConstant. performance_host, MongoDBConstant. cmd_port);} catch (UnknownHostException | unknown exception e) {e. printStackTrace ();} // boolean auth = db. authenticate (myUserName, myPassword);} public UtileMongDB () {try {localBatch = new UtilThreadLocal <ArrayList <DBObject> (ArrayList. class);} catch (Exception e) {e. printStackTrace () ;}}/*** return db object * @ return db */public static DB getDB () {if (db = null) {db = mongo. getDB (MongoDBConstant. MONGO_DB);} return db;}/*** returns mongo * @ return mongo connection pool */public static Mongo getMong () {return mongo ;} /*** read Collection * @ return mongo collection **/public static DBCollection getColl (String collname) {return getDB (). getCollection (collname);} public static DBCollection getColl () {return getDB (). getCollection (MongoDBConstant. MONGO_COLLECTION);}/** crud operation */public void addBatch (String key, String value) {BasicDBObject basicDB = new BasicDBObject (); basicDB. put (key, value);/** use the local variable of the thread here, and there is no need to have a race condition */localBatch. newGet (). add (basicDB);}/*** execute batch processing **/public void executeInsertBatch () {getColl (). insert (localBatch. get (); localBatch. get (). clear ();}/*** execute batch Delete */public void executeDeleteBatch () {ArrayList <DBObject> array = localBatch. get (); for (DBObject obj: array) {getColl (). remove (obj);} localBatch. get (). clear ();} public DBCursor query (String key, String value) {BasicDBObject basicDBObject = new BasicDBObject (); basicDBObject. put (key, value); return getColl (). find (basicDBObject );}}

 

 

ThreadLocal encapsulation:

Package com. saike. solr. server. util; import java. lang. reflect. constructor; /***** @ author xieyong ** @ param <T> the local thread variable object type */public class UtilThreadLocal <T> extends ThreadLocal <T> {/** parameter set */Object [] obj; /** instantiate the Constructor */Constructor <T> construct; /***** @ param clazz class of the local variable * @ param args constructor parameter * @ throws NoSuchMethodException * @ throws SecurityException */public UtilThreadLocal (Class clazz, Obje Ct... args) throws NoSuchMethodException, SecurityException {this. obj = obj; Class [] clazzs = null;/** new gets the parameter class for the constructor */if (args! = Null) if (args. length! = 0) {clazzs = new Class [args. length]; for (int I = 0; I <args. length; I ++) {clazzs [I] = args [I]. getClass () ;}} this. construct = clazz. getConstructor (clazzs);}/*** if no object exists in the current thread, create a new object * @ return */public T newGet () {T tar = super. get (); if (tar = null) {try {tar = construct. newInstance (obj); super. set (tar);} catch (Exception e) {e. printStackTrace () ;}} return tar ;}}

MongoDB batch data insertion failed

It's so sloppy...
// Your I equals 10, which is equivalent to the empty dbobjs set. Of course, no data can be saved .. For (int I = 10; I <10; I ++) {DBObject obj1 = new BasicDBObject (); obj1.put ("name", "I" + I); dbobjs. add (obj1);} is changed to the following. Try again. : For (int I = 0; I <10; I ++) {DBObject obj1 = new BasicDBObject (); obj1.put ("name", "I" + I); dbobjs. add (obj1 );}

How does MongoDB batch insert data?

This operation is not available because the underlying MongoDB does not support this operation.
From the oplog mode of MongoDB, we can see that each op of MongoDB is the smallest unit for independent execution. Therefore, there is no combination of multiple op into one execution. Therefore, this batch insert operation does not exist in various APIs.

In addition, according to my personal experience, my personal experience is correct:
This involves the essential differences between traditional relational databases and NoSQL, which are frequently used. Each NoSQL operation is lightweight and small, with no additional operations except Data Writing.
Analogy: MongoDB simply throws things into the corresponding Cabinet (database) when putting things (data), while MySQL needs to maintain communication with the deliverer (two-way connection ), storage of things in different la S (with mode ).
The batch insert of MySQL reduces the communication and sharding processes, and MongoDB itself does not have these processes. Therefore, MongoDB does not have the concept of batch insert.

Welcome to my blog for further discussion in Reference Materials
Reference: blog.sina.com.cn/u/1448370131
 

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.