Use SBT to compile Spark source in window environment

A few days ago with Maven compiled package spark, make the burn, all kinds of mistakes, endless, think is also drunk, so, in a different way, use SBT compile, see how the character!

First, download the spark source package from the website Spark website and extract it. I'm using the version 1.4.0 on this side.

Then, we need to put SBT configuration, configuration is very simple, is nothing more than sbt_home what, we can refer to the official website of the installation configuration manual.

In the command line mode of window to enter the spark source directory just extracted, we according to the official website prompt Command input:

sbt-pyarn-phadoop-2.3 Assembly

　

Then wait for the compilation to complete ...

Midway Error!!!

Look intently, Oh! It turns out that there is no git command, so I downloaded git from the GIT website and installed and configured the environment variables, which is also very simple.

Continue compiling, the mood is not good, so the command to play longer:

Sbt-pyarn-phadoop-2.6-phive-phive-thriftserver Assembly

A long waiting time ... Take a look at the authoritative guide to Hadoop ...

Failure, failure, and failure!

Back to the point of origin, and turned to maven. I found that Maven was prone to error when compiling the entire spark source code, and it was a bit of a hassle to find it. So, I decided to a small folder compiled, found that really can ah. Now compiling the Pom.xml in the small folder ...

Compile, talk about the root directory of the Pom.xml modification, delete unnecessary module, otherwise, MAVEN compile test time or error, just need to have the rest of the line.

　　

Use SBT to compile Spark source in window environment