front-facing conditions
The configuration of the Hadoop and MySQL database servers has been successfully installed, and if you import the data or export it from hbase, you should also have successfully installed HBase.
Download the JDBC driver for sqoop and MySQL
sqoop-1.2.0-cdh3b4.tar.gz :http://archive.cloudera.com/cdh/3/sqoop-1.2.0-CDH3B4.tar.gz
mysql-connector-java-5.1.28
Installing Sqoop
[Email protected] ~]$ tar -zxvf sqoop-1.2.0-cdh3b4.tar.gz
Configuring environment Variables
Copy
Hadoop
Core Packages and
MYSQL
Drive Package to
Sqoop
of the
Lib
Catalogue
[email protected] ~]$ cp Hadoop-1.1.2/hadoop-core-1.1.2.jar sqoop-1.2.0-cdh3b4/lib/
[email protected] ~]$ cp Mysql-connector-java-5.1.28-bin.jar sqoop-1.2.0-cdh3b4/lib/
Configuring the Sqoop-1.2.0-cdh3b4/bin/configure-sqoop file
Comment out HBase and zookeeper checks (unless you are ready to use Hadoop components such as habase)
start a Hadoop cluster
start MySQL
Create sqoop user
Build the Sqoop Library, test table, and construct the testing data
test Sqoop Connection
[Email protected] ~]$ sqoop list-databases--connect jdbc:mysql://10.120.10.11:3306/--username sqoop--password sqoop
Lists the names of all databases in MySQL
import from MySQL to HDFs
Sqoop # #sqoop command
Import # # indicates an imported
--connect Jdbc:mysql://ip:3306/sqoop # # tell jdbc, connect MySQL URL
--username Sqoop # # user name to connect to MySQL
--password Sqoop # # connection mysql password
--table Test # # # name of the table exported from MySQL
--fields-terminated-by ' \ t ' # # Specify the field delimiter for the rows in the output file
-M 1 # # 1 map jobs used in the copy process
[Email protected] ~]$ sqoop import--connect jdbc:mysql://10.120.10.11:3306/sqoop--username sqoop--password sqoop--ta ble test--fields-terminated-by ': '-M 1
View import results in Hadoop
export from HDFs to MySQL
Sqoop
Export # # means data is copied from hive to MySQL
--connect Jdbc:mysql://ip:3306/sqoop
--username Sqoop
--password Sqoop
--table Test # #mysql table, the name of the table that will be imported
--export-dir '/user/root/aa/part-m-00000 ' # files exported in #hive
--fields-terminated-by ' \ t ' # #hive the delimiter of the exported file field
[Email protected] ~]$ sqoop export--connect jdbc:mysql://10.120.10.11:3306/sqoop--username sqoop--password sqoop--ta ble test--export-dir '/user/hadoop/test/part-m-00000 '--fields-terminated-by ': '-M 1
view export results in Mysql
import from MySQL to Hbase
Parameter description:
Øhbase_tablename Specifies the name of the table to be converted to HBase
Økey_col_name specifies which column in the MySQL database table is the rowkey of the hbase new table
Øcol_fam_name is the column family name for all columns except Rowkey
[Email protected] ~]$ sqoop import--connect jdbc:mysql://10.120.10.11:3306/sqoop--username sqoop--password sqoop--ta ble test--hbase-create-table--hbase-table mysql_sqoop_test--column-family info--hbase-row-key id-m 1
viewing results in HBase
Transferred from: http://www.blogjava.net/redhatlinux/archive/2014/05/31/414291.html
Sqoop installation configuration and data import and export