Crontab+shell Job Flow scheduling __hadoop

In the previous article, we had a simple reference to the pseudo code of the shell or Python dispatch hive, and today we are enriching the pseudocode

http://blog.csdn.net/haohaixingyun/article/details/51821444

Note that in the actual production environment, you may deploy some mature job flow scheduling engine Oozie or Azkaban, and more detailed information please check

Today we have three shell scripts for code, the Code has reference online teaching materials

1. crontab

*/10 * * * * root/opt/main.sh

2,mapreduce_job.sh Script

cmd = "Hadoop jar Example.jar xxx-d ..."

$cmd

Code=$?

while[$code!=0];d o

echo "Run job failed, run submit ..."

$cmd

Done

3,hive_job.sh Script

cmd = "hive-e \" use MyDatabase. ALTER TABLE T add if not EXISTS partition (pt = "20160719") location '/gr/data/20160719;

Select COUNT (*) from t where pt = ' 20160719 '

$cmd

Code=$?

While [$code!=0];d o

echo "Run job failed, run submit ..."

$cmd

Done

4, main.sh to establish a dependency relationship

SH mapreduce_job.sh

SH hive_job.sh