Anatomy Spark-shell

By using Word count to perform the process in Spark-shell, we want to see what Spark-shell did? The following script is in the Spark-shell, see Code Listing 1-1.

Code Listing 1-1 Spark-shell

1234567891011

functionmain() {  if$cygwin; thenstty -icanonmin 1 -echo> /dev/null2>&1    exportSPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -Djline.terminal=unix"    "$FWDIR"/bin/spark-submit--class org.apache.spark.repl.Main "${SUBMISSION_OPTS[@]}"spark-shell "${APPLICATION_OPTS[@]}"sttyicanon echo> /dev/null2>&1  else    exportSPARK_SUBMIT_OPTS    "$FWDIR"/bin/spark-submit--class org.apache.spark.repl.Main "${SUBMISSION_OPTS[@]}"spark-shell "${APPLICATION_OPTS[@]}"fi}

We see the script Spark-shell executed the Spark-submit script, then open the Spark-submit script and discover that it contains the following script.

1	exec"$SPARK_HOME"/bin/spark-classorg.apache.spark.deploy.SparkSubmit "${ORIG_ARGS[@]}"

The script spark-submit adds parameter sparksubmit to the Spark-class script when it executes it. Open the Spark-class script, which contains the following script, as shown in Listing 1-2.

Code Listing 1-2 Spark-class

123456789101112

if [ -n "${JAVA_HOME}"]; then  RUNNER="${JAVA_HOME}/bin/java"else  if[ `command-vjava` ]; then    RUNNER="java"  else    echo"JAVA_HOME is not set">&2    exit1  fifiexec "$RUNNER"-cp"$CLASSPATH"$JAVA_OPTS "[email protected]"

Read this and you should know that spark started the JVM process with Sparksubmit as the primary class.

Note: Spark-shell is a best practice for Linux shells

Anatomy Spark-shell