Big Data Practice-Data Synchronization Chapter Tungsten-relicator (Mysql->mongo)

Reading guidance

With the rapid development of the company's business data volume is also rapidly increasing, based on the user each dimension depth analysis, the relationship data pressure is more and more large; so eager to find some solutions; The research took a long time to finally adopt the Golang+mongod cluster of this scheme, using MONGO to do data analysis of the storage side, Data synchronization becomes a problem, the current online mainstream tools and solutions are relatively few, the only a little more than the article is Tungsten-relicator, the last technology selection is also used it, at present also used for a year, has encountered a lot of problems, but the basic is relatively stable.

Tungsten-relicator Introduction

The tungsten Replicator is a high-performance, open-source data replication engine for MySQL, Postgres, and Oracle databases. This is one of the core components of Continuent's most advanced clustering solution.

Third-party data replication engine--tungsten-replicator key Features:

1 support for high-version MySQL replication to the lower version, 5.1-->5.0
2 support for replication across database systems, Mysql-->pgsql
3 supports multi-master library replication to a single Slave, Multi-master-->slave
4 G-replicator Extract the updated record of data written to the MySQL queue table queue; Based on this queue, other application services can be facilitated

Program Design :

The company used MySQL master and slave, in order not to affect the normal business, but also added a slave library, from the second from the library to the MONGO cluster; This article does not describe the MySQL cluster and Monggo cluster, focusing on tungsten-relicator synchronization and deployment

1, stop the master-slave synchronization from the library, export all the data from the library, emptied from the library;

2. Configure synchronization from the library and the second slave library

3, Build Tungsten-relicator synchronization (Mysql-mongo)

4. Import the data exported from the library from a new library

5. Restart the master/slave synchronization.

Diagram after deployment is complete

Build Tungsten-relicator Synchronization

Tungsten-relicator needs to be deployed to two servers, the primary service is responsible for reading the MySQL Binlog log parsing to the slave server, receiving data from the server and synchronizing to MONGO

first Configure the primary server (192.168.0.1)

1. Install the basic Environment JAVA RUBY

Yum-y Install java-1.7. 0-openjdk*-y install Ruby

2, modify the maximum number of links to the system

1) View Ulimit-n

2) Change

Vim/etc/security/limits.conf

* Soft Nofile 65535

* Hard Nofile 65535

3) Restart Linux

Reboot

3. Modify MySQL Configuration

vi/etc/my.cnf Add binlog_format=1

Simultaneous stop synchronization

Slave stop;

4, Tungsten Main program configuration

TAR-ZXVF tungsten-replicator-2.2.1-403. TAR.GZCD Tungsten-replicator-2.2.1-403start./tools/TPM Install Mysql2mongodb--master=192.168.0.1     --install-directory=/opt/continuent--replication-user=Root--replication-password=root    --enable-heterogenous-master=true     --repl-svc-extractor-filters=Replicate--property=replicator.filter.replicate. Do=Zhongxin--property=replicator.filter.pkey.addcolumnstodeletes=true     --property=replicator.filter.pkey.addpkeytoinserts=true      --start

Master--master server IP address

Replication-user--MYSLQ user name

Replication-password--MySQL password

Property=replicator.filter.replicate.do--Synchronized Database library name

5. View Tungsten Sync Status

/opt/continuent/tungsten/tungsten-replicator/bin/trepctl status

State:online indicates that the service is starting properly

Configuring the Slave server (192.168.0.2)

1. Install the basic Environment JAVA RUBY

Yum-y Install java-1.7. 0-openjdk*-y install Ruby

2, modify the maximum number of links to the system

1) View Ulimit-n

2) Change

Vim/etc/security/limits.conf

* Soft Nofile 65535

* Hard Nofile 65535

3) Restart Linux

Reboot

3, configure password-free login (from tungsten from the server password-free login to the master server)

  Ssh-keygen-t RSA All-way    carriage return  CD. SSH/  cp id_rsa.pub Authorized_keys    Authorized_keys  SCP Authorized_keys [email protected]192.168. 0.2:/root/. SSH  -R. SSH

4, tungsten from the service program configuration

TAR-ZXVF tungsten-replicator-2.2. 1-403. TAR.GZCD Tungsten-replicator-2.2. 1-403 start    ./tools/tungsten-installer--master-slave-a \
--DATASOURCE-TYPE=MONGODB \
--DATASOURCE-PORT=27001 \
--master-host=192.168.0.1 \
--SERVICE-NAME=MYSQL2MONGODB \
--home-directory=/opt/continuent \
--java-file-encoding=utf8 \
--svc-parallelization-type=none \
--start-and-report

MongoDB is installed locally

Master-host--Main service address

5. View Tungsten Sync Status

/opt/continuent/tungsten/tungsten-replicator/bin/trepctl status

State:online indicates that the service is starting properly

6. Start MySQL sync data

Start slave;

Operation Chapter

1. View the log of the Sync tool

tail-300f  /opt/continuent/tungsten/tungsten-replicator/log/-30f/opt/continuent/service_logs/ Trepsvc.log

2. View the status of the synchronization

/opt/continuent/tungsten/tungsten-replicator/bin/trepctl Status/opt/continuent/tungsten/ TUNGSTEN-REPLICATOR/BIN/TREPCTL Services

3, when the synchronization error, after resolving the problem, execute the command resynchronization

/opt/continuent/tungsten/tungsten-replicator/bin/trepctl-Service Mysql2mongodb online/opt/continuent/ Tungsten/tungsten-replicator/bin/trepctl status

4, when some of the tables have special symbols may cause synchronization error, you can start from the server when the parameter skip synchronization of the table

--property=replicator.filter.replicate.ignore=zhongxin.zx_notice_req_log \

If after a period of time, for some reason you need to erase the data resynchronization, you can install the steps

1, stop the master-slave synchronization from the library, export all the data from the library, emptied from the library;

2. Delete MySQL Tungsten_mysql2mongodb library from library

3. Delete MONGO's TUNGSTEN_MYSQL2MONGODB library

4, restart the master-slave synchronization of the tungsten (Installation Start command)

5. Import the data exported from the library from a new library

6, start MySQL master-slave synchronization.

Big Data Practices-Data Synchronization Chapter Tungsten-relicator (Mysql->mongo)