Mosql: simplifies synchronization between MongoDB and PostgreSQL

Recently, the San Francisco-based Stripe Company released mosql, which allows you to easily generate reports from MongoDB using a replicated PostgreSQL database. Mosql is based on mongoriver, which is also a product of stripe. It can monitor MongoDB data updates in almost real time.

Mosql aims to simulate the traditional RDBMS design. Its reports and real-time queries are performed on the read-only copies of product data. Before being provided to business analysts, this read-only copy usually needs to be converted several times, so you should be familiar with this.

Mongoriver

Mongoriver is a common library used for MongoDB oplog tailing. It is written in Ruby. developers can view MongoDB instance update operations through mongoriver. Currently, there are not many mongoriver documents, and the GitHub site only provides the source code. Mongoriver is based on the MIT license.

Mosql

Mosql is built on mongoriver and performs actual data conversion. It requires a yaml-style ing file called "set ing File ". In the preparation phase, developers only need to create the file. Mosql automatically creates the required target table in PostgreSQL.

Mosql can run in one-time or tailing mode. In one-time mode (enabled using the "Skip-tail" flag), it only executes the import once. In tailing mode, it monitors the above oplog, so that it can be synchronized with PostgreSQL. When you start mosql, you can also force import. This operation will delete the current table and create a new table.

If mosql encounters a non-conforming mode Value in the MongoDB database (for example, a floating point value is used in the integer field), it will issue a warning, ignore the entire object, and continue.
If the field in the MongoDB object is not in the Set ing, it will discard the additional field unless: extra_props is set in: Meta hash. If this parameter is set, it collects all discarded fields, encodes them into hash in JSON format, and then stores the generated text in _ extra_props. You can use this JSON for processing. One way is to use plv8 to parse them in PostgreSQL. You can also take out the JSON and parse it in the application code.

Mosql is also based on the MIT license.

View Original English text:Mosql: synchronizing MongoDB and PostgreSQL made easy

View Original Chinese Text: Mosql: simplifies synchronization between MongoDB and PostgreSQL