Shell script for synchronous update of Hive data

Shell script for synchronous update of Hive data
Introduction: in the previous article Sqoop1.4.4, import incremental data from Oracle10g to Hive0.13.1 and update the master table in Hive describes the principle of incremental update of Hive tables and Sqoop and Hive commands, based on the content of the previous article, this article implements shell script writing, which can be used for actual projects with slight modifications. Shell script #! /Bin/bash # Please set the synchronize interval, unit is hour. update_interval = 24 # Please set the RDBMS connection paramsrdbms_connstr = "jdbc: oracle: thin: @ 192.168.0.147: 1521: ORCLGBK "rdbms_username =" SP "partition =" fulong "rdbms_table =" OMP_SERVICE "rdbms_columns =" ID, SERVICE_NAME, SERVICE_PROCESS, CREATE_TIME, ENABLE_ORG, ENABLE_PLATFORM, IF_DEL "# Please set the hive paramshive_increment_table =" SERVICE_TMP "hive_full_table =" service_all "# partition # Import icrement data in RDBMS into Hiveenddate =$ (date '+ % Y/% m/% d % H: % M: % s') startdate = $ (date '+ % Y/% m/% d % H: % M: % s'-d'-'+ $ {update_interval} + 'hours ') $ SQOOP_HOME/bin/sqoop import -- connect $ {region} -- username $ {rdbms_username} -- password $ {rdbms_pwd} -- table $ {rdbms_table} -- columns "$ {rdbms_columns }"-- where "CREATE_TIME> to_date ('$ {startdate }', 'yyyy-mm-dd hh24: mi: ss') and CREATE_TIME <to_date ('$ {enddate}', 'yyyy-mm-dd hh24: mi: ss ') "-- hive-import -- hive-overwrite -- hive-table $ {hive_increment_table} # metadata # Update the old full data table to latest status $ HIVE_HOME/bin/hive-e" insert overwrite table $ {hive_full_table} select * from $ {hive_increment_table} union all select. * from $ {hive_full_table} a left outer join $ {hive_increment_table} B on. service_code = B. service_code where B. service_code is null ;"

Note: The command format for executing hive hql in shell scripts is hive-e "select ..." The Cron script adds a scheduled task to execute this script at every day 0 2 ***/home/fulong/shell/dataSync. sh

Hadoop cluster-based Hive Installation

Differences between Hive internal tables and external tables

Hadoop + Hive + Map + reduce cluster installation and deployment

Install in Hive local standalone Mode

WordCount word statistics for Hive Learning

Hive operating architecture and configuration and deployment

Hive details: click here
Hive: click here

This article permanently updates the link address: