MongoDB data is automatically synced to ElasticSearch

Source: Internet
Author: User
Tags solr mongo shell

Our products require full-text search functionality, and back-end data storage primarily uses MySQL + MongoDB, where the content that needs to be retrieved is in MongoDB.

MongoDB itself is self-featured with text indexing, but it does not support Chinese. The technology industry has specialized, MongoDB is the data storage application, then the full text search uses the specialized full-text search engine bar.

Some of the pre-selected contestants are: SOLR, ElasticSearch, Xapian, Sphinx, Xunsearch. Because of our large amount of data, I feel that now stand alone is a bit too much, MongoDB also began to plan to do shards, then full-text search if the self-distributed skills that is most appropriate. After a series of considerations, finally, we decided to use the ElasticSearch.

Now the back-end program is to write data directly to MongoDB, I do not want to modify the program code, do not want to add and delete data in the MongoDB while adding and deleting data in ElasticSearch. You want to automatically sync to ElasticSearch when data is sent in MongoDB, so you can use the ElasticSearch as quickly as possible.

At first I found the solution to use the ElasticSearch River to synchronize the data, and on GitHub to the MongoDB River plugin: Elasticsearch-river-mongodb. However, then I read this blog on the ElasticSearch official online: "Deprecating Rivers", the official has been in the version after 1.5 abandoned river, for the user's migration, will remain until the 2.0 version.

So, I have to find another solution. Then I found another solution on the Internet: Mongo-connector. This is a tool written by the official MongoDB developer in Python, which currently supports the synchronization of MONGODB data into SOLR, ElasticSearch, MongoDB, and allows users to expand themselves. See the README in the disclaimer, I kind of feel that this is the ensemble nature of the tool, but still hold the mentality decided to try.

Here is the deployment process:

    1. MongoDB must turn on replication set, if it is already open, ignore this step:
      To configure the name of the replica set:mongod --replSet myDevReplSet
      Initialize the replica set in the MONGO shell:rs.initiate()

    2. Install ElasticSearch, ignore this step if you have already installed it.

    3. Install Mongo-connector:
      Install PIP First:

      yum install python-setuptools && easy_install pip

      Install Mongo-connector with PIP:

      pip install mongo-connector
    4. Run Mongo-connector:

      mongo-connector -m 127.0.0.1:27017 -t 127.0.0.1:9200 -d elastic_doc_manager

OK, now, in MongoDB, the data can be added to the ElasticSearch immediately synchronization. In the trial process, Mongo-connector quit two times, one of the disconnection too long did not notice, I had to resynchronize. Still a little unreliable feeling, may also have to write a Guardian program, let Mongo-connector has been able to work in the backstage.

Expand reading:

    • Implement Chinese search with MongoDB + elasticsearch

    • Using MONGO connector and elasticsearch to achieve fuzzy matching

MongoDB data is automatically synced to ElasticSearch

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.