Multiple character set coexistence case sqoop from MySQL import hbase Chinese garbled solve

Recently in doing Binlog log capture synchronization to the data platform. The initial need for Sqoop to initialize data from the Library data table into HBase, the entire process needs to be automated to minimize human intervention. However, for historical reasons, there are two types of database (tables) in the character set format on the line, and the data imported into hbase needs to be stored uniformly in the utf-8 format. Sqoop Direct import, can not control the Chinese characters transcoding work. So we need to make simple changes to the Sqoop source code to support this way.

The general idea is that you can get table-collation of a table from the tables of the Informationschema library by defining an interface. The reason for this is that the character set granularity that we care about is determined to the table level, because some libraries are UTF but the table is latin1. Put the acquired table-collation into the Importjobcontext and put it in the Conf of Hbaseimportjob. It is then set into the Tablecollation attribute of the Tostringputtransformer by Hbaseputprocessor, and finally, during the assembly of the Put object, This property is used to determine if the string data for latin1 and UTF8 needs to be transcoded.

Also note that at the beginning, you need to execute a query "set names UTF8" in the method that gets the connection to the database.

Specific code can refer to: http://url.cn/UmWQUd, full volume search Huanggang, you can see all the changes in the code snippet.

Multiple character set coexistence case sqoop from MySQL import hbase Chinese garbled solve