Five ways to sync data from MongoDB to Elasticsearch

Elastic Search (ES) is a pop-star for recording and analyzing data, Andmongodb are a famous NoSQL database for storing and q uerying data. With our web infrastructure improving, how can we export data from MongoDB to ES for searching or analyzing purposes? There are 5 possible solutions recommended for your choice.

1.synchronized by Web server

We can use the Mongoosastic module for storing-in-both-sides purpose if we use Nodejs as a Web server Container. When one of the document needs to is stored, mongoosastic can commit the changes to both MONGO and ES. As the chart below:

This is the reference link:mongoosastic.

The advantage is this data can be stored in both MONGO and ES simultaneously, and the downside was that overhead Sed in CUD operation efficiency. and inconsistent data might is generated when one type of the DB store failed. And the server framework isn't flexible enough for db migrating. 2.Manually loading data from Mongo to ES

Transporter tool is a good choice to synchronize data once your want to export MONGO the data to another ES server. Transporter also can export data from or to other type of data store. Reference link is:transporter.

It's important to know this transporter synchronizing only once. When the job was done, the transporter comes to its end. 3. Plugin for ES

There is a plugin to es named "Elasticsearch-river-mongodb", and was widely used in ES 1.x, but now River mechanism for E S 2.x is deprecated. Reference link is elasticsearch-river-mongodb. 4. JDBC Input plugin for Logstash

We can take advantage of buffering, inputting, outputting and filtering abilities from Logstash by adding a MONGO input and ES output plugin to get the this job done.

The JDBC Input plugin is one of the choices, but it needs JDBC driver support. As I know there is no well-supported-free JDBC driver for MONGO. Some trial versions can is found in Unity or Simba.

Reference link is:jdbc Plugin for Logstash 5. Mongo-es connector

Mongo-connector is a real-time sync service as a package Ofpython. It creates a pipeline from a MongoDB cluster to one or more target systems. It needs MONGO to run in Replica-set mode, sync data in MONGO to the target then tails the MONGO. It needs a package named "Elastic2_doc_manager" to write data to ES. Process Chart below:

Reference link Is:github or python.

To recapitulate it, it's a must to Remember:mongo replica set, a opened port with IP for ES, using Elastic2_doc_manager If use ES 2.x.

At present, I am not yet ready and any official support in Beats. It'll be in the future.

So that's all, the 5 ways to Mongo-es-sync.

Note: This paper has been transferred from Https://www.linkedin.com/pulse/5-way-sync-data-from-mongodb-es-kai-hao