Install and use Sqoop
1. What is Sqoop?
Sqoop (SQL to Hadoop) is a convenient tool for data migration between traditional databases and Hadoop. It makes full use of the parallel features of MapReduce to accelerate data transmission in batches, so far, Sqoop1 and Sqoop2 have evolved.
Sqoop is a bridge between relational
Step One: Enter the client shell
fulong@fbi008:~$ sqoop.sh Client
Sqoop Home directory:/home/fulong/sqoop/sqoop-1.99.3-bin-hadoop200
Sqoop shell:type ' help ' or ' \h ' for help.
Sqoop:000> Set server--host FBI003--port 12000--webapp
Tags: unable to strong profile node height Apach JDK Install expSqoop is an open source tool that is used primarily in Hadoop (Hive) and traditional databases (MySQL, PostgreSQL ...) Data can be transferred from one relational database (such as MySQL, Oracle, Postgres, etc.) to the HDFs in Hadoop, or the data in HDFs can be directed into a relational database. The Sqoop project began in 2009 as a third-party module for Hadoop, and later, to enable use
Sqoop is an open-source tool mainly used for data transmission between hadoop and traditional databases. The following is an excerpt from the sqoop user manual.
Sqoop is a tool designed to transfer data between hadoop andrelational databases. you can use sqoop to import data from arelational Database Management System
Hue is an open source graphical management tool under the Apache Foundation, developed using the Python language, using the framework of Django. Sqoop is also an open source tool for Apache, developed using the Java language, primarily for data transfer between HDFS and traditional relational databases. These two days in the integration of these two tools, encountered a problem, hereby recorded.The hue version is the 3.9.0,
Test connection usage for Oracle database① Connect to Oracle database, list all databases[[email protected] sqoop] $sqoop list-databases--connect jdbc 10.1.69.173:1521:orclbi--username huangq-por Sqoop list-databases--connect jdbc Racle:thin10.1.69.173:1521:orclbi--username Huangq--password 123456 or Mysql:sqoop list-databases--connectjdbc:mysql://172.19.17.119:3
is Available#export hive_home=3. Drive Jar PackageThe following test shows MySQL as an example, you need to copy the MySQL corresponding driver lib file to the Test data in 4.Mysqlcreatetable ' Demo_blog ' ( ' id ' int (one) notnullauto_increment, ' blog ' varchar (+) NOTNULL,PRIMARYKEY (' id ')) engine= myisamdefaultcharset=utf8; createtable ' Demo_log ' ( ' operator ' varchar (+) NOTNULL, ' log ' varchar (notnull) ENGINE=MyISAMDEFAULTCHARSET=utf8; Insert test Data:insertintodemo_blog (id,
Install and verify the Sqoop installation and verification environment:
System
Redhatlinux 6.4
HadoopVersion
1.2.1
SqoopVersion
1.4.4
MysqlDatabase version
5.6.15
Implement data http://www.linuxidc.com/Linux/2013-06/85817.htm between Mysql/Oracle and HDFS/Hbase through Sqoop
[Hadoop] Sqoop installation proces
Tags: hadoop HDFS sqoop MySQL
Sqoop is a plug-in the hadoop project. You can import the content in HDFS of the Distributed File System to a specified MySQL table, or import the content in MySQL to the HDFS File System for subsequent operations.
Test Environment Description:
Hadoop version: hadoop-0.20.2
Sqoop: sqoop-1
Use sqoop to import data from a MySQL database to hbase
Prerequisites: Install sqoop and hbase.
Download jbdc DRIVER: mysql-connector-java-5.1.10.jar
Copy the mysql-connector-java-5.1.10.jar to/usr/lib/sqoop/lib/
Command for importing hbase from MYSQL:Sqoop import -- connect JDBC: mysql: // 10.10.97.116: 3306/Rsearch -- table researchers -- hbase-Table A -- colum
Deployment Installation # Sqoop is a tool for transferring data from Hadoop and relational databases to each other, and can lead data from a relational database (e.g. MySQL, Oracle, Postgres, etc.) into the HDFs of Hadoop. HDFs data can also be directed into a relational database.# Deploy Sqoop to 13.33, reference documentation: Sqoop installation configuration a
commands view data under HDFsImport the local data file into the HDFs:Compare data to an HDFS path using Hadoop directivesCreate External TableifNot exists EMP (IDintComment'User name', namestringComment'Month', JobstringComment'Number of visits') Comment'User Access Table' row format delimited fields terminated by "\ T" location "/user/hive/warehouse/test.db"; --Method 1. -put/root/part-m-00000 /user/hive/warehouse/test.db--Method 2. "/root/part-m
Recently in the data analysis of a traffic flow, the demand is for a huge amount of urban traffic data, need to use MapReduce cleaning after importing into hbase storage, and then using the Hive External table associated with hbase, hbase data query, statistical analysis, Save the analysis results in a hive table, and finally use Sqoop to import the data from that table into MySQL. The whole process is probably as follows:
Below I mainly
SqoopRelational DB and Hive/hdfs/hbase import the exported MapReduce framework.Http://archive.cloudera.com/cdh5/cdh/5/sqoop-1.4.4-cdh5.1.0/SqoopUserGuide.htmlEtl:extraction-transformation-loading abbreviations, data extraction, transformations (business processing), and loading.File data Source: Hive load CommandRelational DB data Source: Sqoop ExtractionSqoop Import data to hdfs/hive/hbase--> Business proc
Sqoop installation: Installed on a node on the can.1. Upload Sqoop2. Install and configure add SQOOP to environment variable copy the database connection driver to $sqoop_home/lib 3. Use the first class: Data in the database is imported into HDFs SQOOP import--connect jdbc:mysql://192.16 8.1.10:3306/itcast--username root--password 123--table trade_detail--columns
BackgroundSqoop is a tool used to transfer data from Hadoop and relational databases (RDBMS) to each other. When using Sqoop, we need to provide the access password for the database. Currently Sqoop supports 4 ways to enter passwords:
Clear text mode.
Interactive mode.
File mode.
Alias mode.
The author uses the Sqoop in CDH5.10, the vers
;
+----+------+------+ |id|name|age| +----+------+------+ |7|a|1| | 8|b|2| |9|
c|3| +----+------+------+ 3rowsinset (0.00sec) 2. Licensing for individual users Note: After the Sqoop commits the job, each node accesses the database during the map phase, so the prior authorization is required mysql> Grant [All | select | ...] on {db}. {table} to {User}@{host} identified by {passwd};
mysql> flush Privileges;
#我给特定的hostname授权 username:root passwd:root Ac
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.