Hadoop2.6.0 Startup script Analysis

start-all.sh Start principle step1:start-all.sh

First look at the start-all.sh script in the Sbin directory.

Throw comments with little content:

# Start all Hadoop daemons. Run this on master node.

This script is not recommended for use in hadoop-2.6.0. The recommended use is to start start-dfs.sh and start-yarn.sh separately.

Its content steps:

1. Locate the path to the Bin folder and set Hadoop_libexec_dir this environment variable.

2. Execute the hadoop-config.sh script for Hadoop_libexec_dir (actually LIBEXEC under the Hadoop directory) (This script sets environment variables, etc.)

3. Determine if there are Start-dfs and Start-yarn scripts in the Sbin directory. If so, execute the start-dfs.sh and start-yarn.sh two scripts separately,

Run the yarn and HDFs processes (daemon). Open the service.

Where the Hadoop_conf_dir (Hadoop's profile directory usually refers to/etc/hadoop) behind the--config

echo "This script are Deprecated.instead use start-dfs.sh and start-yarn.sh"
 
bin= ' DirName "${bash_source-$0}" '
bin= ' CD ' $bin '; PWD '
 
default_libexec_dir= "$bin"/. /libexec
hadoop_libexec_dir=${hadoop_libexec_dir:-$DEFAULT _libexec_dir}
. $HADOOP _libexec_dir/ hadoop-config.sh
 
# start HDFs daemons if HDFs is present
if [f ' ${hadoop_hdfs_home} '/sbin/start-dfs.sh]; then
   "${hadoop_hdfs_home}"/sbin/start-dfs.sh--config$hadoop_conf_dir
fi
# start yarn daemons if yarn is present
if [f "${hadoop_yarn_home}"/sbin/start-yarn.sh]; then
 "${hadoop_yarn_home}"/sbin/start-yarn.sh-- Config$hadoop_conf_dir
fi

step2:start-dfs.sh

# Start Hadoop Dfs daemons.

# optinally Upgrade or Rollback DFS state.

# Run this on master node.

Start the Hadoop Dfs daemons, which can be followed by parameters. The parameters are rollback and upgrade, and Clusterid.

This script needs to be started on the master node.

Usage= "Usage:start-dfs.sh[-upgrade|-rollback] [other options such as-clusterid]"

How to use: usually do not add anything, direct start-dfs.sh can be

bin= ' dirname ' ${bash_source-$0} '
bin= ' CD ' $bin '; pwd '
 
default_libexec_dir= ' $bin '/... /libexec
hadoop_libexec_dir=${hadoop_libexec_dir:-$DEFAULT _libexec_dir}
. $HADOOP _libexec_dir/ hdfs-config.sh

Like Step1, no longer repeat.

bin= ' dirname ' ${bash_source-$0} '
bin= ' CD ' $bin '; pwd '
 
default_libexec_dir= ' $bin '/... /libexec
hadoop_libexec_dir=${hadoop_libexec_dir:-$DEFAULT _libexec_dir}
. $HADOOP _libexec_dir/ hdfs-config.sh

Determine the parameter if the start-dfs.sh has a condition that is greater than or equal to 1 arguments.

# Get arguments
if [$#-ge 1]; then
            namestartopt= ' "'
            shift case
            ' $nameStartOpt ' in
              (-upgrade)
                     ;;
              (-rollback)
                     datastartopt= "$nameStartOpt"
                     ;;
              (*)
                          echo $usage
                          exit 1
                ;;
            Esac
fi

#Add Other possible options

namestartopt= "$nameStartOpt $@"

Make namestartopt this argument equals $namestartopt $@

(This statement does not execute if there are no arguments.) ）

#---------------------------------------------------------

# namenodes
 
namenodes=$ ($HADOOP _prefix/bin/hdfs getconf-namenodes) echo starting namenodes on[
 
$NAMENODES] "
 
"$HADOOP _prefix/sbin/hadoop-daemons.sh" \
 --config "$HADOOP _conf_dir" \
 --hostnames "$NAMENODES" \
 --script "$bin/hdfs" Start Namenode $nameStartOpt

Start Namenodes

"Namenodes=$ ($HADOOP _prefix/bin/hdfs getconf-namenodes)"

This means setting the NAMENODES environment variable (stand-alone). Where/bin/hdfs is the script that initiates the service and command-line tools associated with HDFs. Because Hadoop is implemented by Java, this script actually adds a Java command line to start the mainclass of services and tools. It is the lowest-level startup script.

Because I am a stand-alone environment, so namenodes is localhost. Because I did not install the zookeeper, just the basic environment. So it only starts with namenodes,datanodes and secondary namenodes.

It should be noted that each module starts the sbin/hadoop-daemons.sh and sets the corresponding properties.

#---------------------------------------------------------

# datanodes (using default slaves file)
 
if [-N "$HADOOP _secure_dn_user"]; then
 echo \
   Attempting to start SE Cure cluster, skipping datanodes. "\
   " Run start-secure-dns.sh as root to complete startup. "
else
 "$HADOOP _prefix/sbin/hadoop-daemons.sh" \
   --config "$HADOOP _conf_dir" \
   --script "$bin/hdfs" start Datanode $dataStartOpt
fi

Because the value of the HADOOP_SECURE_DN_USER environment variable is null (the default is NULL, this safe mode), Echo is not prompted. Instead, use sbin/hadoop-daemons.sh directly to start the datanodes.

#---------------------------------------------------------

# secondary Namenodes (if any)
 
secondary_namenodes=$ ($HADOOP _prefix/bin/hdfsgetconf-secondarynamenodes 2>/ Dev/null)
 
If [-N "$SECONDARY _namenodes"];then
 echo "Starting secondary namenodes [$SECONDARY _namenodes]"
 
 "$HADOOP _prefix/sbin/hadoop-daemons.sh" \
     --config "$HADOOP _conf_dir" \
     --hostnames "$SECONDARY _ Namenodes "\
     --script" $bin/hdfs "start Secondarynamenode
fi

In fact: start-dfs.sh all commands, whether it is to start a single namenode or to start multiple Datanode, is through hadoop-daemons.sh, and then through slaves.sh pushed to the target machine. The only difference is that when the Namenode and Secondary-namenode are started, the push target of the command is shown by the--hostnames parameter (note: The value of the--hostnames parameter is assigned to the HADOOP_ Slave_names This action takes place in the libexec/hadoop-config.sh script, and in hadoop-daemons.sh, the hadoop-config.sh is executed before the call to Slaves.sh, which completes the HADOOP_ Slave_names of the Set value). When the Datanode is started, the--hostnames value is not set, so that the Hadoop_slave_names value is read from the slaves file and the Datanode list.

step3:start-daemons.sh

# secondary Namenodes (if any)
 
secondary_namenodes=$ ($HADOOP _prefix/bin/hdfsgetconf-secondarynamenodes 2>/ Dev/null)
 
If [-N "$SECONDARY _namenodes"];then
 echo "Starting secondary namenodes [$SECONDARY _namenodes]"
 
 "$HADOOP _prefix/sbin/hadoop-daemons.sh" \
     --config "$HADOOP _conf_dir" \
     --hostnames "$SECONDARY _ Namenodes "\
     --script" $bin/hdfs "start Secondarynamenode
fi

The number of parameters after hadoop-daemons.sh must be greater than 1

bin= ' dirname ' ${bash_source-$0} '
bin= ' CD ' $bin '; pwd '
 
default_libexec_dir= ' $bin '/... /libexec
hadoop_libexec_dir=${hadoop_libexec_dir:-$DEFAULT _libexec_dir}
. $HADOOP _libexec_dir/ hadoop-config.sh

The old routine ... (notice here that the value of the--hostnames parameter is assigned to the Hadoop_slave_names in the hadoop-config.sh and this action occurs in the libexec/hadoop-config.sh script.) ）

(The figure is hadoop-config.sh to judge the parameters behind the--hostnames and assign values.) ）

bin= ' dirname ' ${bash_source-$0} '
bin= ' CD ' $bin '; pwd '
 
default_libexec_dir= ' $bin '/... /libexec
hadoop_libexec_dir=${hadoop_libexec_dir:-$DEFAULT _libexec_dir}
. $HADOOP _libexec_dir/ hadoop-config.sh

In hadoop-daemons.sh, the hadoop-config.sh is executed before the slaves.sh is invoked, and the Hadoop_slave_names is completed.

Borrow a diagram:

Excerpt from: http://blog.csdn.net/bluishglc/article/details/43237289

Where: The parentheses in Figure 4:

The environment variable hadoop_conf_dir is etc/hadoop, the following slaves is localhost under the stand-alone environment. On-line environment: content is

In my stand-alone environment, the content is localhost. (online environment is the cluster of my internship company) Step 4 hadoop-daemon.sh

The following is the start daemons, which starts the process. SSH connects each datanode machine and CD to the $hadoop_prefix of each machine (on my sd0 host is/usr/local/hadoop).

Also take this sentence to analyze, has already carried out the completion is slaves.sh. The following to be performed is hadoop-daemon.sh.

1 The value of several environment variables is explained in the annotation.

2 parameter: If the parameter is less than or equal to 1, then tell the usage and exit. Then set the value of the Bin,hadoop_libexec_dir and execute the hadoop-config.sh. (In this order, each script executes the following code before it is formally executed.)

For this script, the function of the hadoop-config.sh script is to handle the files of the configuration file and host list and set the corresponding environment variables to save them.

3 Next, set the Hadoopscript (followed by a nohup reference) and save the startstop variable and command variable.

4 then defines a function called Hadoop_rotate_log (): The Hadoop scrolling log function. It means:

① First Judge Hadoop-deamon with the fifth parameter (according to the previous shift number, if-script = $, otherwise the third parameter) is greater than 0, if so, then the NUM variable is no longer equal to 5, but is equal to the fifth parameter, otherwise it is 5.

② $log (fourth parameter) is a file, if it is, then scroll the log (rotate logs)

③ Order is: OUT4->OUT5,OUT3->OUT4,OUT2-OUT3,OUT1->OUT2,OUT->OUT1

5 below, determine if the hadoop-env.sh exists, if there is a execution of this file. This file contains Java_home and other important environment variables, for large clusters, this document is very important. The content in the red box behind it is to set some environment variables. (Take Hadoop_log_dir, first to determine whether it is empty, and then determine whether it has permission to write operations.) )

6 finally use a CASE–ESAC to control the start and stop.

Case $startStop in (start) <strong> if the third/five argument is start, execute the following statement. </strong> [-W "$HADOOP _pid_dir"] | |
Mkdir-p "$HADOOP _pid_dir" <strong> if Hadoop_pid_dir does not have write permission (not writable), create. </strong> if [f $pid]; Then if kill-0 ' cat $pid ' >/dev/null 2>&1; Then kill-0 is to test whether the specified PID exists, 2>&1 stderr and stdout are directed together/dev/null.
       If it does, you are prompted to turn it off before you can start and exit the shell.  Echo $command running as process ' cat $pid '.
       Stop it. Exit 1 fi fi if ["$HADOOP _master"!= "];
     Then echo rsync from$hadoop_master rsync is a data mirroring Backup tool under Unix-like systems, and--remote sync can be seen from the name of the software. Rsync-a-essh--delete--exclude=.svn--exclude= ' logs/* '--exclude= ' contrib/hod/logs/* ' $HADOOP _master/' $HADOOP _
   PREFIX "fi <strong> log file backup </strong> hadoop_rotate_log $log echo starting $command, logging to $log CD "$HADOOP _prefix" case $command in Namenode|secondarynamenode|datanode|journalnode|dfs|dfsadmin|fsck|balancer|zk FC) If [-Z "$HADOOP _hdfs_home"]; ThEn hdfsscript= "$HADOOP _prefix"/bin/hdfs else hdfsscript= "$HADOOP _hdfs_home"/bin/hdfs fi &l The T;strong>nohup command is to continue the task after logging out of the remote computer, and the Nice command is used to start another program with the specified process scheduling priority. Reference from: Http://man.linuxde.net/nice </strong> nohup nice-n $HADOOP _niceness $hdfsScript--config$hadoop_conf_di
     R $command "$@" > "$log" 2>&1 </dev/null &;  (*) nohup nice-n$hadoop_niceness $hadoopScript--config $HADOOP _conf_dir $command "$@" > "$log" 2>&1 <
   /dev/null &; Esac Echo $! > $pid <strong>$! ID </strong> sleep 1 head "$log" # capture Theulimit output <s for this shell's last running process Trong> the output of the restricted relationship between the file system and the program </strong> if ["true" = "$starting _secure_dn"]; Then echo "Ulimit-a for secure Datanode user$hadoop_secure_dn_user" >> $log # Capture info for the A Ppropriate user <strong>su– is not only switching users ' identities, but also switching the shell environment.  With the shell environment switched, the path's environment variables are valid (~/.bash_profile or ~/.profile) </strong>   Su--shell=/bin/bash $HADOOP _secure_dn_user-c ' ulimit-a ' >> $log 2>&1 elif [' true ' = ' $starting _privil Eged_nfs "]; Then echo "Ulimit-a for privileged NFS User$hadoop_privileged_nfs_user" >> $log Su--shell=/bin/bash $ Hadoop_privileged_nfs_user-c ' ulimit-a ' >> $log 2>&1 else echo "ulimit-a for USER $USER" >> $log ul Imit command. Ulimit the resources used by the shell startup process.
     –A displays all current resource restrictions.
Ulimit-a >> $log 2>&1 fi 3; <strong> if the shell's last process ID does not exist, exit the shell </strong> if! Ps-p $! >/dev/null;
 Then exit 1 fi;; (stop) If [-F $pid]; Then target_pid= ' Cat $pid ' <strong> first determine if the process exists, if present, the first if is the normal kill process, if failed, "Sleep" hadoop_stop_timeout time. The second if uses kill-9 to forcibly kill the process. (Kill-l can view all process communication signals) </strong> if kill-0 $TARGET _pid >/dev/null 2>&1;  Then echo stopping $command kill $TARGET _pid sleep $HADOOP _stop_timeout if kill-0 $TARGET _pid >/dev/null 2>&1;
Then         echo "$command did not stop gracefully after $HADOOP _stop_timeoutseconds:killing with kill-9" kill-9 $TARGET _pid fi Else echo no $command to the Stop fi rm-f $pid else echo no $command to
 
  Stop fi;;
 
(*) echo $usage exit 1;;
 Esac

In general, the execution hadoop-env.sh sets the environment variable because the JVM that is about to start is started by this shell, so this environment variable is passed to the JVM.

Configure the operating environment for startup Point Namenode or Datanode: In addition to hadoop-config.sh inside, there are hadoop_log_dir,hadoop_pid_dir,hadoop_ident_string and so on, These are variables that are related to the native computer running this daemon.

Finally through

Start the corresponding process, which is the HDFs start Namenode command. It's actually invoking the Bin/hdfs script to start the JVM.

hadoop-daemon.sh This script is the preparation for starting a variety of JVMs on each machine, including setting environment variables and starting the process. Here you should pay special attention to the problem of invalidation of environment variables after SSH to each node. (Excerpt from: http://www.cnblogs.com/zhwl/p/3670997.html)

step5:start-yarn.shyarn.config.sh

This script is to start all the yarn processes.

Notice that inside the script, There is no direct execution hadoop-config.sh setting the environment variable, but rather by executing the yarn-config.sh (① hadoop-config.sh within yarn-config.sh, if hadoop-config.sh in three environment variables Libex EC directory to implement (avoid the problem of environmental variables set up chaos.) ) to set the environment variable if there is no error in the three directories.

② then introduces the MALLOC_ARENA_MAXZ environment variable. The default is 4, which can be set to the value you want. If you set a smaller value, you limit the number of memory arenas and virtual memory, and (in a non-large build environment) do not significantly degrade performance. GLIBC in order to allocate memory performance problems, using a number of memorypool called Arena, the default configuration under 64bit is each arena 64M, a process can have a maximum of cores * 8 arena. Assuming your machine is 4 cores, then you can have a maximum of 4 * 8 = 32 Arena, which means using the =2048m memory. (Excerpt from: http://blog.csdn.net/chen19870707/article/details/43202679.) There is no in-depth understanding of this piece. ）

③ Check that the yarn-config is followed by the argument that the first parameter is-config and--hosts. (corresponding, respectively, set the Yarn_conf_dir and yarn_slaves these two environment variables.) )

Finally, the implementation of ResourceManager, NodeManager and ProxyServer. of which, ResourceManager and ProxyServer are initiated through yarn-daemon.sh, NODEMANAGER.SH is initiated through yarn-daemons.sh.

step6:yarn-daemon.shyarn-daemons.sh

similar to hadoop-daemon.sh and hadoop-daemons.sh, also performs yarn-config.sh (setting environment variables, etc.). )

Similar to hadoop-daemon.sh, but hadoop_pid_dir and other environmental variables into the yarn_pid_dir.

Here's the final code for yarn-daemon.sh (I didn't put the stop in here, the main analysis is start.) )

Case $startStop in (start) if the third/five argument is start, execute the following statement. [-W "$HADOOP _pid_dir"] | |
   Mkdir-p "$HADOOP _pid_dir" is created if Hadoop_pid_dir does not have write permission (not writable). If [f $pid]; Then if kill-0 ' cat $pid ' >/dev/null 2>&1; Then kill-0 is to test whether the specified PID exists, 2>&1 stderr and stdout are directed together/dev/null.
       If it does, you are prompted to turn it off before you can start and exit the shell.  Echo $command running as process ' cat $pid '.
       Stop it. Exit 1 fi fi if ["$HADOOP _master"!= "];
     Then echo rsync from$hadoop_master rsync is a data mirroring Backup tool under Unix-like systems, and--remote sync can be seen from the name of the software. Rsync-a-essh--delete--exclude=.svn--exclude= ' logs/* '--exclude= ' contrib/hod/logs/* ' $HADOOP _master/' $HADOOP _ PREFIX "fi log file backup Hadoop_rotate_log $log echo starting $command, logging to $log CD" $HADOOP _prefix "case $ command in NAMENODE|SECONDARYNAMENODE|DATANODE|JOURNALNODE|DFS|DFSADMIN|FSCK|BALANCER|ZKFC) if [-Z "$HADOOP _h Dfs_home "]; Then hdfsscript= "$HADOOP _prefix"/bin/hdfs else hdfsscript= "$HADOOp_hdfs_home "/bin/hdfs fi nohup command is to continue the task after logging out of the remote computer, the Nice command is used to start other programs with the specified process scheduling priority. From: Http://man.linuxde.net/nice nohup nice-n $HADOOP _niceness $hdfsScript--config$hadoop_conf_dir $command "$@" & Gt
     "$log" 2>&1 </dev/null &;  (*) nohup nice-n$hadoop_niceness $hadoopScript--config $HADOOP _conf_dir $command "$@" > "$log" 2>&1 <
   /dev/null &; Esac Echo $!  > $pid $! The ID of the last-run process for this shell Sleep 1 head "$log" # capture Theulimit output to obtain a limit of the file system and program of the export if ["true" = "$starting _secure_dn"]; Then echo "Ulimit-a for secure Datanode user$hadoop_secure_dn_user" >> $log # Capture info for the A Ppropriate user su– is not only switching users ' identities, but also switching the shell environment. With the shell environment switched, the environment variable for path is valid (~/.bash_profile or ~/.profile) su--shell=/bin/bash $HADOOP _secure_dn_user-c ' ulimit-a ' >& Gt $log 2>&1 elif ["true" = "$starting _privileged_nfs"]; Then echo "Ulimit-a for privileged NFS User$hadoop_privileged_nfs_user ">> $log su--shell=/bin/bash $HADOOP _privileged_nfs_user-c ' ulimit-a ' >> $log 2&G T;&1 else echo "ulimit-a for user $USER" >> $log ulimit command. Ulimit the resources used by the shell startup process.
     –A displays all current resource restrictions.
Ulimit-a >> $log 2>&1 fi 3; If the shell's last process ID does not exist, exit the shell if! Ps-p $! >/dev/null;
 Then exit 1 fi;; (stop) If [-F $pid]; Then target_pid= ' Cat $pid ' first determine if the process exists, if it exists, the first if is the normal kill process, if the failure, "sleep" hadoop_stop_timeout time. The second if uses kill-9 to forcibly kill the process. (Kill-l can view all process communication signals) if kill-0 $TARGET _pid >/dev/null 2>&1;  Then echo stopping $command kill $TARGET _pid sleep $HADOOP _stop_timeout if kill-0 $TARGET _pid >/dev/null 2>&1; Then echo "$command did not stop gracefully after $HADOOP _stop_timeoutseconds:killing with kill-9" Kil L-9 $TARGET _pid fi else echo no $command to stop fi rm-f $pid else echo no $comman
  D to stop fi;;
 
(*) echo $usage exit 1;;
 Esac

The analysis here is basically the same as hadoop-daemon.sh, no longer repeat. But here's a difference: in start-dfs.sh, you start a process by calling Hadoop-daemon.sh in the hadoop-daemons.sh. And start-yarn.sh is divided into yarn-daemon.sh and yarn-daemons.sh to start ResourceManager and Nodemanger respectively.

# start NodeManager
"$bin"/yarn-daemons.sh--config $YARN _conf_dir  startnodemanager
(start-yarn.sh)
 
# Run A Yarn command on the all slave hosts.
Exec "$bin/slaves.sh"--config$yarn_conf_dir cd "$HADOOP _yarn_home" \; $bin/yarn-daemon.sh "--config $YARN _conf_dir" $@ "
(yarn-daemons.sh)

Similar to the boot Datanode, it is also done by SSH to each slave node after the execution of the yarn-daemon.sh boot corresponding NodeManager. (Note that setting the appropriate environment variables is also recommended here)

Summary:

in general, the whole start-up process is done from top to bottom. Configuration from the entire cluster level, the configuration of a single machine, and the configuration of a single JVM. See the flowchart below for details.