Datax data synchronization between HDFs and MySQL

This case only applies to the data synchronization implementation between HDFs and MySQL.

1, before compiling the installation note will release the following Setsharepath method in the path to your own installation path, the author of the/home/woody/datax/libs.

2, RPM packaging, to modify the path in the *.spec file, Dataxpath, in addition, you may also need to comment out some of the assignment file code, such as Cp–r ${oldpwd}****.properties ****.conf, The jar copy statements in the Engine.spec are also commented out.

3, note that the Hdfsreader/writer (the path of the two files also need to be changed) in the replication Hadoop-core. The jar package in the statement is changed to your own Hadoop version of the Hadoop-core.jar package.

4, Mysqlreader/writer only need to modify the path.

5. The following issues are mainly related to Field\line by conformance:

Let's do it in the normal order:

MySQL data is first synchronized to HDFs, which requires Mysqlreader and Hdfswriter

Mysqlreader file does not need to be modified

field_spilt= ' \u0001 ' in Hdfswriter file

line_spilt= ' \ n ', other places do not need confirmation, only need to confirm these two places.

The HDFS data is then synced to MySQL, which requires hdfsreader and Mysqlwriter

Hdfsreader file field_spilt= ' \ t ', this confirmation can be

sep= ' \001 ' in Mysqlwriter file (note that the item remains the same)

String sql= "LOAD DATA LOCAL INFILE ' Hdfs://localhost:9000/test_in/part '"

Sql+=string.format ("Dields TERMINATED by ' \u0001 ' escaped by ' \ \ '")

Sql+=string.format ("LINES TERMINATED by ' \ n '")

Also note that the sep= ' \u0001 ' in the Mysqlwriterinputstreamadapter.java file

break = ' \ n ' line delimiter, null= ' \\n ' (NULL literal character) These two items are also worth noting.

The above items can be packaged and installed after the confirmation is complete.

Next you can start synchronizing data from MySQL to HDFs, to configure jobs under the Mysqlreader_to_hdfswriter.xml file:

Ip\port\dbname\username\password\table these are configured themselves, where value= "1=1";

SQL Value= "SELECT * from Test_datax"; Columns value= "*" These three items can be confirmed,

Hadoop.job.ugivalue= "Woody,supergroup"; hadoop_conf value Write your own Hadoop core-site.xml file path, dir value= "hdfs://localhost:9000/test_in"; field_spilt value= "\u0001" ; Line_spilt value= "\ n"; Nullchar value = "\ n" These items are confirmed.

You can then execute the data Synchronization command.

Next you can start synchronizing data from HDFs to MySQL, to configure jobs under the Hdfsreader_to_mysqlwriter.xml file:

Hadoop.job.ugi\hadoop_conf\dir\field_spilt the same as above, nullstring value= "\ n";

Col_filter a comment out.

Ip\port\dbname\username\password\table these are the same as above, pre value= "select * from Test_datax"; Post value= "select * fromtest_datax"; Colorder value= "User_id,user_name";

Set value= "User_id=user_id,user_name=user_name"

You can then execute the data Synchronization command.

Datax data synchronization between HDFs and MySQL