Construction of Spark Source Code and Application Development Environment

Construction of Spark Source Code and Application Development Environment

Debugging and reading Spark Source Code is undoubtedly an effective way to thoroughly learn the Internal principles of Spark. Based on the author's hands-on practices, this article quickly builds a Spark Source Code development and debugging environment based on common development tools, this helps readers quickly learn from Spark.

Introduction

Spark is undoubtedly one of the most popular technologies in the big data field. Readers can easily find articles about how to apply Spark technology. However, as a developer, after learning about the concept of application, the habit is to open the development environment and develop some applications to learn more deeply about spark. When encountering problems, I also hope to debug the Spark source code in depth to solve the problem. However, due to the newer Spark technology, this problem will always occur for beginners in the process of building a development and debugging environment. Compared with the Java language and Maven build tool, the Scala language and SBT build tool are also relatively small, because some information that can be referenced on the Web is relatively small. Based on the author's practice, this article starts to compile the entire project from the Spark source code, and provides some problems that may occur during the compilation process for your reference. The compilation methods shown in each step in this article are mainly used to facilitate the learning of Spark Source Code. If you only need to write a Spark application, you do not need such a complicated process. For detailed reference methods, refer to the last chapter.

Environment requirements

1. System: Windows/Linux/Mac OS
2. IDE: Eclipse/IntelliJ
3. Other dependencies: Scala, Sbt, Maven

Configuration of the Development and debugging environment in Eclipse

The tools used in this section are Windows 7 + Eclipse Java EE 4.4.2 + Scala 2.10.4 + Sbt 0.13.8 + Maven3.3.3, And the Spark version tested is 1.4.0.

1. Configure IDE:

1. Select the menu item Help-> Install new software and add the site http://download.scala-ide.org/sdk/lithium/e44/scala211/stable/site. choose to Install Scala IDE for Eclipse and Scala IDE Plugins.

Figure 1. Configure the IDE

Figure 1. Configure the IDE

For the standard version of Eclipse, you also need to install a separate Maven plug-in.

For ease of configuration, you can also use the Scala IDE officially provided by Scala that has packaged all dependencies.

In particular, because the Project itself has some errors, please temporarily disable the Project-> Build Automatically function to save time.

2. Download The Spark Source Code:

Create an empty directory and execute the following statement: git clone https://github.com/apache/spark.git

In addition to using git commands, You can also download the packaged source code from the Spark Github page.

3. Convert the source code to an Eclipse project:

Go to the source code root directory and execute the following statement: sbt eclipse. Sbt downloads all jar packages required by Spark during execution, so this step takes a long time. Some jar packages can be downloaded only by using network proxy. After the execution is successful, there will be 2 outputs.

Figure 2. Convert to Eclipse Project

4. Import the project to Eclipse:

Select the menu item File> Import, and select General> Existing Projects into Workspace. Select the source code root path for the project root path and Import all the Projects (a total of 25 Projects ).

5. Modify Scala version:

Go to Preference-> Scala-> Installations and add Scala 2.10.4 installed on the machine (select the lib directory ). Because this version of Spark (1.4.0) is written in Scala 2.10.4 environment, you need to modify the Scala version used by the project in Eclipse. Method: select all projects, right-click Scala, select Set the Scala Installation, and select the corresponding Scala version. 3.

Figure 3. Modify Scala version

6. Add Scala Library for the old-deps project:

Right-click the old-deps project and choose Scala> Add Scala Library to Build Path.

7. Maven install to generate the classes required by spark-streaming-flume-sink:

First copy the scalastyle-config.xml file in the source code root directory to the spark-streaming-flume-sink project root directory, then open the project in Eclipse, right-click the pom. select "Run as-> Maven install" for the xml file. 4.

Figure 4. Maven install

After running successfully, the console outputs 5 contents.

Figure 5. Maven install successful

8. An error occurred while modifying the spark-SQL and spark-hive packages:

Because the source code package settings are incorrect, you need to move the class file to the correct package. 6.

Figure 6. package modification Error

For the spark-SQL project, select test.org in src/test/java. apache. spark. SQL and test.org. apache. spark. SQL. all classes in the sources package, right-click and choose Refactor-> Move to org. apache. spark. SQL and org. apache. spark. SQL. sources package.

For the spark-hive project, select test.org in src/test/java. apache. spark. SQL. all classes in hive and test.org.apache.spark. SQL .hive.exe cution package are moved to org. apache. spark. SQL. hive and org.apache.spark. SQL .hive.exe cution packages.

-------------------------------------- Split line --------------------------------------

Spark1.0.0 Deployment Guide

Install Spark0.8.0 in CentOS 6.2 (64-bit)

Introduction to Spark and its installation and use in Ubuntu

Install the Spark cluster (on CentOS)

Hadoop vs Spark Performance Comparison

Spark installation and learning

Spark Parallel Computing Model

-------------------------------------- Split line --------------------------------------

9. compile all projects:

Open the Project-> Build Automatically function and wait until all projects are compiled successfully.

10. Check whether the installation is successful:

Copy the src-> main-> resources-> org folder in the core project to target-> scala-2.10-> classes in the examples project. Then execute the org. apache. spark. examples. SparkPi program in the examples project and set its jvm parameter to-Dspark. master = local. If the last 7 results are output, the installation is successful.

Figure 7. SparkPi running result

For more details, please continue to read the highlights on the next page:

1 2 Next Page