Shell script completes cluster installation of Hadoop

Although the overall implementation of the automatic installation, but there are many needs to improve the place, such as:

1. The code can only be run under the root authority, otherwise it will be wrong, this need to add permission to judge;

2. You can also add several functions to reduce code redundancy;

3. Some of the judgements are not intelligent enough;

......

The ability and the time are limited, can only write here.

The Installhadoop file code is as follows:

#!/bin/bash#root_password= "123456" jdk_tar=jdk-8u65-linux-i586.tar.gzjdk_url=http://download.oracle.com/otn-pub /java/jdk/8u65-b17/jdk-8u65-linux-i586.tar.gzjdk_version=jdk1.8.0_65java_version=1.8.0_65jdk_install_path=/usr /local/developmenthadoop_url=http://101.44.1.4/files/2250000004303235/mirrors.hust.edu.cn/apache/hadoop/common /stable1/hadoop-1.2.1.tar.gzhadoop_version=hadoop-1.2.1hadoop_tar=hadoop-1.2.1.tar.gzhadoop_install_path= hadoophadoop_tmp_path=/home/hadoop/hadoop_tmphadoop_name_path=/home/hadoop/hdfs/namehadoop_data_path=/home/ HADOOP/HDFS/DATAUSER_NAME=HADOOPUSER_PASSWD=HADOOP#SU #判断能否root # If [$-ne 0]; Then#echo "No root access" #exit # fishfilepath=$ (PWD) #check jdk installed or notjava-version &>/dev/nullif [$?-ne 0]; Thenecho {JDK has been INS Talled in this pc}java-versionelse# check ~/. /usr/local/development directory exists No, does not exist to create # first into the current user's home directory #cd ~ #cd: /.. /usr/local/$jdk _install_path &>/dev/null#if [$?-ne 0]; then if [!-D $jdk _install_path]; Thenecho "{Create $jdk _install_path folder to install JDK} "mkdir $jdk _install_pathcd $jdk _install_pathecho" {Success to create $JDK _i Nstall_path folder} "Elseecho" {$jdk _install_path folder has already exists} "CD $JDK _install_pathfi# check JDK for decompression #ls | grep "$jdk _version" &>/dev/nullif [!-d $jdk _version]; then# Check if the JDK has a compressed package if [!-f $JDK _tar]; Thenecho "{Downlo Ad $jdk _tar} "wget--no-check-certificate--no-cookies--header" Cookie:oraclelicense=accept-securebackup-cookie "$ Jdk_urlfiecho "{Untar $jdk _tar}" tar-zxvf $jdk _tarelseecho "{$jdk _version folder have already exists in $JDK _install_path/ } "Fi#set jdk Environmentecho {set Java ENVIRONMENT}CD ~cd.. /.. /.. /.. /etc/profile.d/touch $jdk _install_path.sh#echo "#!bin/bash" > $jdk _install_path.shecho "Export java_home=/usr/ local/$jdk _install_path/$jdk _version ">> $jdk _install_path.shecho" Export jre_home=\ $JAVA _home/jre ">> $ Jdk_install_path.shecho "Export classpath=.:\ $JAVA _home/lib:\ $JRE _home/lib:\ $CLASSPATH ">> $jdk _install_path.shecho "path=\ $JAVA _home/bin:\ $JRE _home/bin:\ $PATH" >> $jdk _install_path.sh source $jdk _install_path.sh #check the Java versionjava-version | grep "$java _version" &>/dev/nullif [$-ne 0]; Thenecho ' {Success to install $jdk _version} ' fifi#no passwd when L Ogin via Ssecho "{Config SSH service and login via SSH without no passwd}" sudo yum-y install SSH openssh-server#update/e tc/ssh/sshd_config#rsaauthenticationrsaauthentication_linenum= ' awk '/rsaauthentication yes/{print NR} ' ~/. /etc/ssh/sshd_config ' rsaauthentication= "rsaauthentication yes" sed-i "${rsaauthentication_linenum}s/^.*/${ Rsaauthentication}/g "~/. /etc/ssh/sshd_config#pubkeyauthenticationpubkeyauthentication_linenum= ' awk '/pubkeyauthentication yes/{print NR} ' ~/.. /etc/ssh/sshd_config ' pubkeyauthentication= "pubkeyauthentication yes" sed-i "${pubkeyauthentication_linenum}s/^.*/ ${pubkeyauthentication}/g "~/. /etc/ssh/sshd_config#authorizedkeysfileauthorizedkeysfile_linenum= ' awk '/authorizedkeysfile/{prinT NR} ' ~/. /etc/ssh/sshd_config ' authorizedkeysfile= ' authorizedkeysfile. Ssh\/authorized_keys "Sed-i" ${AuthorizedKeysFile_ Linenum}s/^.*/${authorizedkeysfile}/g "~/. /etc/ssh/sshd_configecho "{You sshd_config as follow}" Sed-n "${rsaauthentication_linenum},${ Authorizedkeysfile_linenum}p "~/. /etc/ssh/sshd_config#restart sshd service~/. /sbin/service sshd Restartecho ' {Finish to update sshd_config} ' #generate public keyif [!-D ~/.ssh]; Thenmkdir ~/.sshfic D ~/.sshecho y | Ssh-keygen-t Rsa-p '-f id_rsaif [!-f Authorized_keys]; Thentouch authorized_keyscat id_rsa.pub > Authorized_keys Elsecat id_rsa.pub >> authorized_keysfichmod ~/.sshchmod ~/.ssh/authorized_keys#download hadoopcd ~cd. /home/$hadoop _install_path &>/dev/nullif [$?-ne 0]; Thenecho "{create/home/$hadoop _install_path folder to Inst All JDK} "CD. /homemkdir $hadoop _install_pathcd $hadoop _install_pathecho "{Success to create $ $hadoop _install_path folder}" Elseecho "{/home/$hadoop _insTall_path folder has already exists} "CD ~CD. /home/$hadoop _install_pathfi#check hadoop-2.7.0 folder is exists or Notif [!-d "$hadoop _version"]; Then#check hadoop-2. 7.0.tar.gz is exist or Notif [!-F "$hadoop _tar"]; Thenecho "{Download $hadoop _tar}" wget--no-check-certificate--no-coo Kies--header "Cookie:oraclelicense=accept-securebackup-cookie" $hadoop _urlfiecho "{Untar $hadoop _tar}" TAR-ZXVF $ Hadoop_tarelseecho "{$hadoop _version folder have already exists in/home/$hadoop _install_path/}" fi#enter into config FOLDERCD $hadoop _versionif [!-D "Conf"]; THENCD etc/hadoop/elsecd conffi#update hadoop-env.shjava_home_line_num= ' awk ' /export java_home/{print NR} ' hadoop-env.sh ' javahome= "Export java_home=\/usr\/local\/" $jdk _install_path "\" $jdk _ Version#-i is directly modify the source filesed-i "${java_home_line_num}s/^.*/${javahome}/g" Hadoop-env.shcat hadoop-env.sh | grep "Java_home" echo "{Finish to update hadoop-env.sh}" hadoop_config_path=$ (pwd) #echo $cur _path#echo $shFilePath # UnaliAs Cp#cp-rf core-site.xml $curPath/cd $shFilePath #update core_site.xmlcat core-site.xml > $hadoop _config_path/ Core-site.xmlif [!-d $hadoop _tmp_path]; thenmkdir $hadoop _tmp_pathfirm-rf $hadoop _tmp_path/*if [!-D $hadoop _name_pat h]; Thenmkdir $hadoop _name_pathfichmod g-w $hadoop _name_pathrm-rf $hadoop _name_path/*if [!-d $hadoop _data_path]; then mkdir $hadoop _data_pathfichmod g-w $hadoop _data_pathrm-rf $hadoop _data_path/* #update mapred-site.xmlcat Mapred-site.xml > $hadoop _config_path/mapred-site.xml#update hdfs-site.xmlcat hdfs-site.xml > $hadoop _config_ PATH/HDFS-SITE.XMLCD $hadoop _config_pathecho "{Check core-site.xml}" #cat Core-site.xmlecho "{Check mapred-site.xml}" #cat Mapred-site.xmlecho "{Check hdfs-site.xml}" #cat hdfs-site.xmlecho "{Finish config Hadoop}" #add Hadoop account and Ha  S admin accessid $user _nameif [$?-ne 0]; Thenecho "{Add $user _name}" sudo useradd-mr $user _namefi#set passwd for Hadoop Accountecho $user _passwd | sudo passwd--stdin $user _nameecho "{Format Hadoop}" Echo Y |.. /bin/hadoop Namenode-formatcd. /bin/bash Stop-all.shecho "{Start Hadoop}" Bash start-all.shresult= ' JPS | awk ' {print $} ' | Xargs ' expect_result= "Jobtracker NameNode DataNode tasktracker Jps secondarynamenode" If ["$result" = = "$expect _result"] ; Thenecho "{Congratulations!!! Success to Intall hadoop!} " Elseecho "{Sorry, fail to install Hadoop and try to restart hadroop!}" Bash Stop-all.shecho "{Start Hadoop}" Bash start-all.shresult= ' JPS | awk ' {print $} ' | Xargs ' If ["$result" = = "$expect _result"]; Thenecho "{Sorry, fail to find all Java thread and please check!}" Elseecho "{Congratulations!!! Find all Java thread, success to install hadoop!} " Fifiecho {!!! Finish!!!}

　　

In addition, to automate the configuration of Hadoop, Core-site.xml, Hdfs-site.xml, and mapred-site.xml files need to be placed in the same directory as the Installhadoop files.

Core-site.xml file:

<?XML version= "1.0"?><?xml-stylesheet type= "text/xsl" href= "configuration.xsl "?><!--Put Site-specific property overrides in this file. -<Configuration>        < Property>            <name>Fs.default.name</name>            <value>hdfs://localhost:9000</value>        </ Property>    < Property>          <name>Hadoop.tmp.dir</name>          <value>/home/hadoop/hadoop_tmp</value>         <Description>A base for other temporary directories.</Description>    </ Property></Configuration>

Hdfs-site.xml:

<?XML version= "1.0"?><?xml-stylesheet type= "text/xsl" href= "configuration.xsl "?><!--Put Site-specific property overrides in this file. -<Configuration>< Property>      <name>Dfs.name.dir</name>      <value>/home/hadoop/hdfs/name</value>  </ Property>  < Property>      <name>Dfs.data.dir</name>      <value>/home/hadoop/hdfs/data</value>  </ Property>   < Property>    <name>Dfs.replication</name>    <value>1</value></ Property></Configuration>

Mapred-site.xml

<?XML version= "1.0"?><?xml-stylesheet type= "text/xsl" href= "configuration.xsl "?><!--Put Site-specific property overrides in this file. -<Configuration>    < Property>        <name>Mapred.job.tracker</name>        <value>localhost:9001</value>    </ Property></Configuration>

A single version of Hadoop installation can be consulted:

Http://www.linuxidc.com/Linux/2015-04/116447.htm

Hadoop installations for multiple machines can be consulted:

http://blog.csdn.net/ab198604/article/details/8250461

Shell script completes cluster installation of Hadoop