Shell script Practice Notes (10)-read the spark cluster script fragments

Preface:

Through studying spark cluster scripts, take notes on some important shell script skills.

*). Get the directory of the current script

sbin=`dirname "$0"` sbin=`cd "$sbin"; pwd`

Code comments:
# The above code is a common technique for getting the directory where the execution script is located
# Sbin = $ (dirname $0) the returned result may be a relative path, such ./
# Sbin = $ (CD $ sbin; PWD) Use PWD to return the absolute path of the directory where the script is located.

*). Loop traversal Script Parameters

while (( "$#" )); do　　case $1 in　　　　--with-tachyon)　　　　　　TACHYON_STR="--with-tachyon"　　　　;;　　esac　　shiftdone

Code comments:
# This is a common code snippet for Traversing script parameters.
# $ # In shell scripts indicates the number of parameters
# Because $0 is occupied by the script name itself, the traversal of the parameter of the script starts from $1. With the help of shift variable left shift, the traversal of the variable-length parameter list is convenient.
# Event-based XML parsing. When pull is used to traverse data, it is similar to the code structure.
# Of course, it should be noted that after shift processes the parameter variables, it will have an impact (negative effect) on subsequent script code processing variables. Therefore, the best practice is to process the Script Parameters in a centralized manner.

*). Introduce the configuration script.

. "$sbin/spark-config.sh"

Code comments:
# In shell scripts, '.' is equivalent to source. The call script is executed as a part of the caller's script. source <shell_file> is usually used to import application configuration parameters.
# Differences between source/exec/fork external scripts

*). Default parameter Processing

if [ "$SPARK_MASTER_PORT" = "" ]; then　　SPARK_MASTER_PORT=7077fi

Code comments:
# Processing Method for default variable values
# Note: When you add "", if [$ spark_master_port = ""], an error is returned: "[: =: unary operator expected"
# Similar code can adopt the-Z $ spark_master_port Method

if [ -z $SPARK_MASTER_PORT ]; then　　SPARK_MASTER_PORT=7077fi

*). $! And PID File Usage

nohup nice -n $SPARK_NICENESS "$SPARK_PREFIX"/bin/spark-class $command "[email protected]" >> "$log" 2>&1 < /dev/null &newpid=$!echo $newpid > $pid

Code comments:
# Nohup indicates that the process runs out of the session
# Nice-N is used to adjust the nice value of the process
#2> & 1: associate a standard error (stderr, 2) with a standard output (stdout, 1), which can be abbreviated as &>
# $! PID of the previous Shell Command (run in the background)
# Echo $ newpid> $ PID (representing the file), which writes the process PID to the PID file of the process.
# Many services (such as APACHE) choose to write their own PID (process ID) into the PID file. Why? Each has its own application scenarios. The following kill-0 is applied.

*). Use kill-0 to check whether the process exists and re-import (misjudgment) the problem

if [ -f $pid ]; then　　if kill -0 `cat $pid` > /dev/null 2>&1; then　　　　echo $command running as process `cat $pid`. Stop it first.　　　　exit 1　　fifi

Code comments:
# Kill-0 <pid> simply sends a signal to the process (which does not affect the process running) to check whether the process exists. (echo $? => 0), does not exist (echo $? => 1)
# If [-F $ pid] determines whether the PID file exists. If cat $ PID is used, the PID value is obtained, which is consistent with the PID File above.
# Kill-0 'cat $ PID '>/dev/null 2> & 1 followed by'>/dev/null 2> & 1' to remove unnecessary information to the console
Question:
# Re-entry problem: A bit similar to TCP problem, socket occupies the four tuples (SRC: IP + port, DEST: IP + port), the legacy TCP packet, interference caused by subsequent re-use of port socket
# Hypothesis: After the PID is written to the PID file, the process exits and a new process occupies the PID, therefore, it makes no sense for the script to determine whether the previous process is alive based on this PID, which leads to misjudgment.
# Linux kernel uses a latency redistribution policy for PID allocation. If the PID is reused, it will lead to re-determination.

*). Concurrency + wait usage

for slave in `cat "$HOSTLIST"|sed "s/#.*$//;/^$/d"`; do　　ssh $SPARK_SSH_OPTS $slave $"${@// /\\ }" 　　　　2>&1 | sed "s/^/$slave: /" &　　if [ "$SPARK_SLAVE_SLEEP" != "" ]; then　　　　sleep $SPARK_SLAVE_SLEEP　　fidonewait　

Code comments:
# Shell scripts do not have the concept of multithreading, and the execution of sub-shells is blocked by default. Therefore, you can only run multiple sub-processes in the background to simulate
# SSH $ slave "<command>" & is to run the SSH command in the background
# Wait: waits until all background processes are finished before continuing.
# This is a good programming practice for concurrent countdownlatch.

*) Use the. Sed command

sed "s/#.*$//;/^$/d"sed "s/^/$slave: /"

Code comments:
# Use the stream editor sed to replace and delete text content. Like SED