The text of this text connection is: http://blog.csdn.net/freewebsys/article/details/47722393 not allowed to reprint without the Bo master.1, about SqoopSqoop is a tool that transfers data from Hadoop and relational databases to each other, and can import data from a relational database such as MySQL, Oracle, Postgres, etc. into Hadoop's HDFs. You can also import HDFs data into a relational database.Official website: http://sqoop.apache.org/A 1.4.6 version, one is 1.99 version (development versi
1. Install the deployment(1), download address: http://archive.cloudera.com/cdh5/cdh/5/sqoop-1.4.6-cdh5.5.2.tar.gzUnzip to/home/neil/downloads/hadoop-2.7.3/sqoop-1.4.6(2), copy the MySQL JDBC driver package Mysql-connector-java-5.1.31-bin.jar to the Sqoop/lib directory.(3), configuration environment variable sqoop
Expo
Background of sqoopMost enterprises that use hadoop technology to manage Big Data businesses have a large amount of data stored in traditional relational databases (RDBMS; due to lack of tool support, it is very difficult to transmit data between hadoop and traditional database systems. sqoop is a project for data transmission between RDBMS and hadoop;
Sqoop Overview
S
Welcome to the big Data and AI technical articles released by the public number: Qing Research Academy, where you can learn the night white (author's pen name) carefully organized notes, let us make a little progress every day, so that excellent become a habit!First, Sqoop's introduction:Sqoop is a data acquisition engine/data exchange engine that captures database in relational databases (RDBMS) primarily for data transfer between RDBMS and Hdfs/hive/hbase and can be
Sqoop is a tool used for data import and export, typically used in the framework of Hadoop, where common scenarios include importing data from a MySQL database into HDFs or hive, Hbase, or exporting it to a relational database. The following sections show the process of importing and exporting several pieces of code.
Import data from MySQL into the Hadoop cluster (HDFS):
The script command is first posted:
./sqo
1. List all databases in MySQL database
Sqoop list-databases--connect jdbc:mysql://localhost:3306/-username Dyh-password 000000
2. Connect MySQL and list the tables in the database
Sqoop list-tables--connect jdbc:mysql://localhost:3306/test--username dyh--password 000000
3. Copy the table structure of the relational data into hive
Sqoop create-hive-table--connect
The error tolerance of Sqoop itself depends on Hadoop. Here we focus on the processing of Sqoop transmission task failure. Specifically, how does the focus solve the data consistency problem caused by the failure of the transmission task in Sqoop?
For A transfer task, data is transmitted from A to B. If the transfer task fails, the statuses of A and B should be c
Tags: sqoop1.997Link: Http://pan.baidu.com/s/1gfHnaVL Password: 7j12Mysql-connector version 5.1.32If you encounter some problems during the installation process, refer to http://dbspace.blog.51cto.com/6873717/1875955, some of which are the solution to the problemDownload and install:cd/usr/local/TAR-ZXVF sqoop2-1.99.3-cdh5.0.0.tar.gzMV sqoop2-1.99.3-cdh5.0.0 SqoopAdd SQOOP2 to the system environment variable:Export Sqoop_home=/usr/local/sqoopExport catalina_base= $
Use sqoop to import mysql Data to hadoop
The installation and configuration of hadoop will not be discussed here.Sqoop installation is also very simple. After sqoop is installed, you can test whether it can be connected to mysql (Note: The jar package of mysql should be placed under SQOOP_HOME/lib): sqoop list-databases -- connect jdbc: mysql: // 192.168.1.109: 3
master HBase Enterprise-level development and management• Ability to master pig Enterprise-level development and management• Ability to master hive Enterprise-level development and management• Ability to use Sqoop to freely convert data from traditional relational databases and HDFs• Ability to collect and manage distributed logs using Flume• Ability to master the entire process of analysis, development, and deployment of Hadoop complete projects
Use Sqoop to import MySQL Data to Hadoop
The installation and configuration of Hadoop will not be discussed here.Sqoop installation is also very simple. After Sqoop is installed and used, you can test whether it can be connected to mysql (Note: The jar package of mysql should be placed under SQOOP_HOME/lib ): sqoop list-databases -- connect jdbc: mysql: // 192.16
Tags: mysql hive jdbc Hadoop sqoopThe installation configuration of Hadoop is not spoken here.The installation of Sqoop is also very simple. After you complete the installation of SQOOP, you can test if you can connect to MySQL (note: The MySQL Jar pack is to be placed under Sqoop_home/lib): SQOOP list-databases--connect jdbc:mysql://192.168.1.109:3306/--username
Tags: exporting. NET size Data Conversion ref DIR username Nat tmpHive Summary (vii) hive four ways to import data (strongly recommended to see) Several methods of data export of Hive https://www.iteblog.com/archives/955 (strongly recommended to see) Import MySQL data into HDFs 1. Manually import using MySQL tools The simplest way to import MySQL's exported data into HDFs is to use command-line tools and MySQL statements. To export the contents of the entire data table or the entire database
Data synchronization between a relational database and a non-Relational Database
1. Without sqoop
MySQL --> hive
1. Use naivacat (Tool) to export tables in the database (primary TAB/T should be used for export)
2. Use winscp (Tool) to upload data to a specified Linux directory
3. Create Table in hive first (idfa string) Row format delimited fields terminated by '\ t '"
4. hive-e "load data local inpath 't1.txt 'into Table T1"
(If there is data in the
1, import incremental imports of the official description2, test sqoop increment ImportIncremental import in the enterprise, generally need to execute frequently, such as one weeks to perform an incremental import, so the incremental import method needs to be executed several times, and each execution, and write the corresponding execution command, it is more troublesome. And Sqoop provides a great tool for
If you specify n as the line break imported by sqoop, if the value of a string field in mysql contains n, sqoop imports an additional line of records. There is an option-hive-drop-import-delimsDropsn, r, and1fromstringfieldswhenimportingtoHive.
If you specify \ n as the line break for sqoop import, if the value of a string field in mysql contains \ n, it will le
1. Import data from MySQL into hiveSqoop Import--connect Jdbc:mysql://localhost:3306/sqoop--direct--username root--password 123456-- Table Tb1--hive-table tb1--hive-import-m 1Where--table tb1 is a table in the MySQL sqoop database,--hive-table tb1 is the name of the table that was imported into hive without having to build the table beforehand.2. Import data from hive into MySQLSqoop export--connect jdbc:my
1. Environment Preparation 1.1 software versionsqoop-1.4.52. ConfigurationThe configuration of the Sqoop is relatively simple, and the files that need to be configured are given below2.1 Environment variablessudo VI /etc/profilesqoop_home=/home/hadoop/source/sqoop-1.4. 5 PATH= $SQOOP _home/binexport sqoop_home2.2sqoop-env.sh#Set Path towhereBin/hadoop isAvailable
Note: The following information refer to the teacher Dylan
What is a sqoop?
Sqoop is an open source tool, Sqoop SQL to Hadoop, used primarily in Hadoop (Hive) and traditional databases (MySQL, PostgreSQL ...) Data transfer, the development of the main evolution of the two major editions, SQOOP1 and SQOOP2.
Second Why Choose S
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.