Python script uses Sqoop to import MySQL data into hive

Turn: 53064123

Using Python to import data from the MySQL database into hive, the process is to manipulate sqoop in Python language.

#!/usr/bin/env python#Coding:utf-8# --------------------------------#Created by Coco on 16/2/23# ---------------------------------#Comment: Main function Description: Initialize the business databaseImportOSImportPyhs2conn=pyhs2.connect (host="192.168.8.94", port=10000,authmechanism="PLAIN", user="HDFs") Mysql_info={"Host":"192.168.8.94","Port": 3306,"User":"Root","passwd":"gc895316"}PrintMysql_infodefrun_hive_query (SQL): With Conn.cursor () as Cursor:cursor.execute (SQL)returnCursor.fetchall ()defmysql_to_hive (host,port,user,passwd,database,table):#Os.system ("Hadoop fs-rm-r/user/task/%s"%table)    if[Database] not inchRun_hive_query ("Show Databases"): With Conn.cursor () as Cursor:cursor.execute ("CREATE DATABASE"+database) with Conn.cursor () as Cursor:cursor.execute (" Use"+database)if[Table] not inchRun_hive_query ("Show Tables"): Os.system ("sqoop Import--connect jdbc:mysql://%s:%s/%s--username%s--password%s--table%s--hive-database%s-m Ten--create-hive-table--hive-import--hive-overwrite"%(host,port,database,user,passwd,table,database))Else: Os.system ("sqoop Import--connect jdbc:mysql://%s:%s/%s--username%s--password%s--table%s--hive-database%s-m 1 0--hive-import--hive-overwrite"%(Host,port,database,user,passwd,table,database)) mysql_to_hive (mysql_info["Host"],mysql_info["Port"],mysql_info["User"],mysql_info["passwd"].replace ("(","\("),"WWN","CM_VIP")

Python script uses Sqoop to import MySQL data into hive