Use IntelliJ IDEA to import the latest Spark Source Code and compile Spark Source Code

Use IntelliJ IDEA to import the latest Spark Source Code and compile Spark Source Code

After some experience in Spark, in order to be able to follow up the development progress of Spark Source code, the source code is read and analyzed in detail, this article details how to use IntelliJ IDEA to import the latest Spark Source code from Github and compile it.

Preparations

First, JDK 1.6 + and Scala must be installed in your system. After downloading the latest IntelliJ IDEA, install the Scala plug-in first (we recommend that you install it when you open it for the first time. Now, Scala can be run in the command line in your system. My system environment is as follows:

1. Mac OS X (10.9.5)

2. JDK 1.7.71

3. Scala 2.10.4

4. IntelliJ IDEA 14

In addition, we recommend that you first use pre-built Spark to understand the running and usage of Spark, compile some Spark applications, and then read the source code, and try to modify the source code for manual compilation.

Import the Spark project from Github

Open IntelliJ IDEA, select VCS → Check out from Version Control → Git in the menu bar, enter the address of the Spark Project in the Git Repository URL, and specify the local path, as shown in.

Click Clone in the window and start to clone the project from Github. This process may take 3-10 minutes to determine the network speed.

Compile Spark

After cloning, IntelliJ IDEA will automatically prompt you whether the project has a corresponding pom. xml file and whether to open it. Select Open pom. xml file, and the system will automatically parse the dependencies of the project. This step will also take a different time because of your network and system environment.

After this step is completed, manually edit the pom under the Spark root directory. xml file, find the line with the specified java version (java. version), according to your system environment, if you are using jdk1.7, you may need to change its value to 1.7 (the default value is 1.6 ).

Then open the shell terminal, enter the root directory of the spark project just imported in the command line, and execute

Sbt/sbt assembly

This compilation command will all use the default configuration to compile Spark, if you want to specify the version of the relevant components, you can view the Spark official site of Build-Spark (http://spark.apache.org/docs/latest/building-spark.html), view all the common compilation options. Currently, this process can be completed without a VPN. to estimate the time required for compilation, you can open a new shell terminal and constantly check the size of the spark project directory. In the end, I use the default configuration, the spark directory size after compilation is 2.0 GB.

Conclusion

So far, to verify your compilation results, you can enter the spark/bin directory in the command line and run spark-shell. If everything starts properly, the compilation is successful. If you modify the Spark source code, you can use sbt to compile it again, and the compilation time is not as long as the first compilation. If you have any questions, please comments!

-------------------------------------- Split line --------------------------------------

Spark1.0.0 Deployment Guide

Install Spark0.8.0 in CentOS 6.2 (64-bit)

Introduction to Spark and its installation and use in Ubuntu

Install the Spark cluster (on CentOS)

Hadoop vs Spark Performance Comparison

Spark installation and learning

Spark Parallel Computing Model

-------------------------------------- Split line --------------------------------------

Spark details: click here
Spark: click here

This article permanently updates the link address: