Spark Source Code Analysis (a)--spark-shell analysis

1. Prepare for Work 1.1 install spark and configure spark-env.sh

You need to install spark before using Spark-shell, please refer to http://www.cnblogs.com/swordfall/p/7903678.html

If you use only one node, you can not configure the slaves file, the spark-env.sh file needs only to be configured for MASTER_IP and local_ip two properties

spark-env.sh Add the following configuration:

Export spark_master_ip=hadoop1export spark_local_ip=HADOOP1

Note: The HADOOP1 is the IP address of this virtual machine, or it can be used 127.0.0.1 instead of HADOOP1. Spark-shell analysis is based on the spark-2.2.0-bin-hadoop2.7 version.

1.2 Start Spark-shell

Enter the Spark installation directory under the bin to execute the Spark-shell command

cd/opt/app/spark-2.2.0-bin-hadoop2.7/bin/
. /spark-shell

Finally, we'll see the spark START process:

2. Execute the Word count sample

Using the word count example to feel the execution of the spark task, after starting Spark-shell, open the Scala command line and follow the steps below to enter the script.

1) Enter val lines = Sc.textfile (". /readme.md ", 2)

2) Enter val words = lines.flatmap (line = Line.split (""))

3) input val ones = words.map (W = (W, 1))

4) Enter val counts = Ones.reducebykey (_ + _)

5) Input Counts.foreach (println)

3. Anatomy Spark-shell

Check out what Spark-shell did by using Word count to perform the process in Spark-shell. The following script is in the Spark-shell

We saw that the script Spark-shell executed the Spark-submit script, opened the Spark-submit script, and found that it contained the following script:

The script spark-submit adds parameter sparksubmit to the Spark-class script when it executes it. Open the Spark-class script, which contains the following script:

Read here, Spark-class inside the first load spark-env.sh inside the configuration properties, and then get the JDK Java command, and then get spark_home jars directory. At this point, Spark initiates a JVM process with the Sparksubmit primary class.

To facilitate the use of remote monitoring of the spark process locally, append the following JMX configuration to the Spark_home directory conf/spark-defaults.conf configuration file:

  #driver端监控spark. driver.extrajavaoptions  =-xx:+unlockcommercialfeatures -xx:+flightrecorder-dcom.sun.management.jmxremote - Dcom.sun.management.jmxremote.port=10207 - Dcom.sun.management.jmxremote.authenticate=false - Dcom.sun.management.jmxremote.ssl=false  # Executor-end monitoring, temporarily annotated #spark.executor.extrajavaoptions  =-xx:+unlockcommercialfeatures-xx:+flightrecorder- Dcom.sun.management.jmxremote 
-dcom.sun.management.jmxremote.port=0 - Dcom.sun.management.jmxremote.ssl=false - Dcom.sun.management.jmxremote.authenticate=false  

Note : Before using remote monitoring, if the Spark-shell need to stop before running, configure the monitoring parameters, and then need to run the Spark-shell command, otherwise JVISUALVM cannot find the thread. the above two commands are one line, not two lines, two rows cause JVISUALVM to connect , reported "unable to use Service:jmx:rmi:///jndi/rmi://192.168.187.201:8009/jmxrmi Connect to 192.168.187.201:8009 "error.

In the local java_home/bin directory, open JVISUALVM, add a remote host, right-click the added remote host, add a JMX connection,

Click the Threads tab on the right, select the main thread, and then click the Thread Dump button.

Find the thread main information from the contents of the dump,

Main thread Dump information

From the stack information of the main thread, you can see the sequence of calls to the program: Sparksubmit.main-Repl. Iloop.process, Main. The Org.apache.spark.repl.SparkILoop class inherits the Iloop class, and the Iloop process method invokes the LoadFiles (settings) and Printwelcome () methods of Sparkiloop. Sparkiloop's LoadFiles (settings) method calls its own Initializespark method, and the Initializespark implementation is as follows:

Initializationcommands is a collection of commands, see Code:

As you can see from the code, the Org.apache.spark.repl.Main createsparksession () method is called in the command collection to create the Sparksession class.

From the above code can see Builder is sparksession inside the property, the idea tool using "CTRL + mouse click" Operation, you can enter into the Builder.getorcreate () method inside to see how sparksession created,

From the code above you can see that Sparkcontext was first created and then created Sparksession. The creation code for Sparkcontext is as follows:

The initialization is done here using sparkconf, Sparkcontext, and Sparksession, and the repl involved in code analysis is used primarily to interact with spark in real time.

At this point, Spark-shell parsing is complete.

Spark Source Code Analysis (a)--spark-shell analysis