Sqoop Import relational database-Decrypt Sqoop

Sqoop as a Hadoop The bridge between the traditional database and the data import and export plays an important role. by expounding the basic grammar and function of Sqoop, the paper deeply decrypts the function and value of Sqoop.

First, what is Apache Sqoop?

Clouderadeveloped byApacheOpen Source project, isSql-to-hadoopthe abbreviation. Mainly used inHadoop (Hive)with the traditional database(MySQL,PostgreSQL ...)data can be transferred to a relational database (for example,:MySQL, Oracle, Postgresand so on) data into theHadoopof theHDFS, you can also addHDFSdata into a relational database. When importing data, you can import either an entire database or some data from a single table, all tables, or tables toHDFS, the data can be imported into various formats at the same time. In addition,Sqoopcan be fromHDFSExport the data to the database, soSqoopIt has a bidirectional effect.

650) this.width=650; "title=" 11.png "alt=" wkiol1hsnb6xycnuaacgs2da1sq677.png-wh_50 "src=" http://s3.51cto.com/ Wyfs02/m02/8b/95/wkiol1hsnb6xycnuaacgs2da1sq677.png-wh_500x0-wm_3-wmp_4-s_339375399.png "/>

Second, Sqoop How does it work?

Sqoop work is also based on the computational framework MapReduce,mapreduce will put the submitted SQL The derivative is converted into MapReduce job, and then commit to the cluster. In general, it is divided into three steps:

1. Check Table details

2. Create and submit jobs to the cluster

3. get table records and write data to HDFS

650) this.width=650; "title=" 22.png "alt=" wkiom1hsnc6zfjjuaab_7eqgiy4318.png-wh_50 "src=" http://s4.51cto.com/ Wyfs02/m01/8b/99/wkiom1hsnc6zfjjuaab_7eqgiy4318.png-wh_500x0-wm_3-wmp_4-s_2617177207.png "/>

Three, basic grammar:

Sqoopis a command-line tool with many subcommands, with many tools for importing, exporting, and listing database content. Usually we first list which tables the database has, and then go to the corresponding tables. If you want to seeSqoopall the tools that can runSqoop Helpto see a list of all the tools here, it's also recommended that you get used to Helpto find some usage parameters. Alternatively, you can also runSqoop help Tool-nameto view Help for a specific tool , such asImport,Export).

Sqoop basic syntax for calls

650) this.width=650; "title=" 33.png "alt=" wkiol1hsnd3d8wlbaaaosb8omm8795.png-wh_50 "src=" http://s3.51cto.com/ Wyfs02/m01/8b/95/wkiol1hsnd3d8wlbaaaosb8omm8795.png-wh_500x0-wm_3-wmp_4-s_677443395.png "/>

Iv. Use of Sqoop Explore the database:

Usually we have to explore the database before conducting the data to understand which tables are in the database, generally there are two ways:

1 , List-tables : Lists all MySQL in Loudacre Table of the database

650) this.width=650; "title=" 44.png "alt=" wkiom1hsneyhvd7vaab6b7tlzho733.png-wh_50 "src=" http://s3.51cto.com/ Wyfs02/m00/8b/99/wkiom1hsneyhvd7vaab6b7tlzho733.png-wh_500x0-wm_3-wmp_4-s_1317998959.png "/>

2.eval : Executing a database query

650) this.width=650; "title=" 55.png "alt=" wkiom1hsnfqd_bo7aacbjq_gmu4964.png-wh_50 "src=" http://s2.51cto.com/ Wyfs02/m02/8b/99/wkiom1hsnfqd_bo7aacbjq_gmu4964.png-wh_500x0-wm_3-wmp_4-s_900453609.png "/>

Sqoop as a Hadoop The important part of the family, we need to focus on cognition and grasp, for our understanding and learning Big data has important help. You can also focus on some big data information, understand the trends of big data development and some industry classic cases. Especially now the network media flourished, information dissemination quickly, such as service number, I usually like to find some cases from above, like big data cn is very good. But technology is always a threshold, only continuous learning and communication can continue to progress, recently I in the Big Data Times Learning Center This service number also learned some other people to share the experience, for my personal help is also very large.

This article is from the "11872756" blog, please be sure to keep this source http://11882756.blog.51cto.com/11872756/1883001

Sqoop Import relational database-Decrypt Sqoop