Bulk upload of files to HDFs shell scripts

In the Hadoop data mining project, our first step is to get the source data, that is, to put the corresponding data into HDFs, so that Hadoop can calculate, manually upload files to HDFs, it is too time-consuming and laborious, so we could take a framework like Flume, or use a shell script to upload the file. The following mainly provides the general wording of the shell script, for reference only, can be modified according to different business.

Version 1:

#!/bin/bash#set JavaEnvExport Java_home=/export/servers/Jdkexport jre_home=${java_home}/Jreexport CLASSPATH=.:${java_home}/lib:${jre_home}/Libexport PATH=${java_home}/bin: $PATH #set HadoopEnvExport Hadoop_home=/export/servers/Hadoopexport PATH=${hadoop_home}/bin:${hadoop_home}/sbin: $PATH # directory where log files are stored log_src_dir=/export/software/#日志文件上传到hdfs的根路径hdfs_root_dir=/data/clicklog/20151226/#读取日志文件的目录 to determine if there are files that need to be uploadedls$log _src_dir | whileRead FileName Do        if["Hadoop.log1"="$fileName"]; ThenHadoop FS-put $log _src_dir$filename $hdfs _root_dirfi Done

Version 2:

#版本1的问题:
#虽然上传到Hadoop集群上了, but the original file is still there. How to deal with?
#日志文件的名称都是xxxx. log1, upload the file again, because HDFs already exists, will be an error. How to deal with?

#如何解决版本1的问题
# 1. Move the uploaded files to the directory you want to upload first
#2, when moving files to the directory to be uploaded, name the file in a certain format
#/export/software/hadoop.log1/export/data/click_log/xxxxx_click_log_{date}

#!/bin/bash#set JavaEnvExport Java_home=/home/hadoop/app/jdk1.7. 0_51export Jre_home=${java_home}/Jreexport CLASSPATH=.:${java_home}/lib:${jre_home}/Libexport PATH=${java_home}/bin: $PATH #set HadoopEnvExport Hadoop_home=/home/hadoop/app/hadoop-2.6.4Export PATH=${hadoop_home}/bin:${hadoop_home}/sbin: $PATH#日志文件存放的目录log_src_dir=/home/hadoop/logs/log/#待上传文件存放的目录log_toupload_dir=/home/hadoop/logs/toupload/#日志文件上传到hdfs的根路径hdfs_root_dir=/data/clicklog/20170419/#打印环境变量信息Echo "envs:hadoop_home: $HADOOP _home"#读取日志文件的目录 to determine if there are files that need to be uploadedEcho "Log_src_dir:"$log _src_dirls$log _src_dir | whileRead FileName Do    if[["$fileName"= = access.log.*]]; Then    # if["Access.log"="$fileName"]; Then        Date=`Date+%y_%m_%d_%h_%m_%S ' #将文件移动到待上传目录并重命名 #打印信息Echo "moving $log _src_dir$filename to $log _toupload_dir"Xxxxx_click_log_$filename"$date"        MV$log _src_dir$filename $log _toupload_dir"Xxxxx_click_log_$filename"$Date#将待上传的文件path写入一个列表文件willDoingEcho$log _toupload_dir"Xxxxx_click_log_$filename"$Date>> $log _toupload_dir"willdoing."$Date    fi     Done#找到列表文件willDoingls$log _toupload_dir |grepwould |grep-V"_copy_"|grep-V"_done_"| whileRead Line Do#打印信息Echo "toupload is in file:"$line #将待上传文件列表willDoing改名为willDoing_COPY_MV$log _toupload_dir$line $log _toupload_dir$line"_copy_"#读列表文件willDoing_COPY_的内容 (one to upload file name), where line is the path to the list of files to be uploadedCat$log _toupload_dir$line"_copy_"| whileRead Line Do#打印信息Echo "puting. $line to HDFs path ... $hdfs _root_dir"Hadoop FS-put $line $hdfs _root_dir Done        MV$log _toupload_dir$line"_copy_"$log _toupload_dir$line"_done_" Done

Timed Script Execution:

*/1sh /export/servers/shell/uploadfile2hdfs.v2. SH */1sh source/etc/profile. sh /export/servers/shell/uploadfile2hdfs.v1. SH  -L

Bulk upload of files to HDFs shell scripts