Spark1.4.1 Compilation and Installation

1. Download:

Http://spark.apache.org/downloads.html

Select Download Source

2. Source code compilation

1) Unzip

TAR-ZXVF spark-1.4.1.tgz

2. Compiling

Go to the root directory and compile with make-distribution.sh.

CD spark-1.4.1

sudo./make-distribution.sh--tgz--skip-java-test-pyarn-phadoop-2.2-dhadoop.version=2.2.0-phive- Phive-thriftserver-dskiptests Clean Package

If there is an error in the middle, please re-run, try several times, generally can succeed.

After the compilation succeeds, its installation file is under the root directory:

Spark-1.4.1-bin-2.2.0.tgz

3. Installation

Omitted, as in previous versions, will not be written.

4, error problemCluster startup issues:

1) Problem 1:worek node cannot start

Localhost:starting Org.apache.spark.deploy.worker.Worker, logging to/home/lib/spark-1.4.1/sbin/. /logs/org.apache.spark.deploy.worker.worker-1-is Xxxx.out

Localhost:failed to launch Org.apache.spark.deploy.worker.Worker:

Localhost:at Org.apache.spark.launcher.SparkClassCommandBuilder.buildCommand (sparkclasscommandbuilder.java:98)

Localhost:atorg.apache.spark.launcher.Main.main (main.java:74)

Localhost:full log in/home/lib/spark-1.4.1/sbin/. /logs/org.apache.spark.deploy.worker.worker-1-is Xxxx.out

Localhost:connection to localhost closed.

The reason is that the system comes with Java problems

Rpm-qa | grep java

Gcc-java-4.4.7-4.el6.x86_64

Java_cup-0.10k-5.el6.x86_64

java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64

Unloading

RPM-E--nodeps java_cup-0.10k-5.el6.x86_64

Rpm-e--nodepsjava-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64

2) Issue 2:java_home is not set

Localhost:starting Org.apache.spark.deploy.worker.Worker, logging to/home/lib/spark-1.4.1/sbin/. /logs/spark-org.apache.spark.deploy.worker.worker-1-is Xxxx.out
Localhost:failed to launch Org.apache.spark.deploy.worker.Worker:
Localhost:java_home is not set
Localhost:full log in/lib/spark-1.4.1/sbin/. /logs/org.apache.spark.deploy.worker.worker-1-isxxxx.out
Localhost:connection to localhost closed.

Find the wrong shell file and join the export java_home= ... Can

Spark-env.sh, join the export java_home= ... Can

After the successful start of the interface:

Reprint please specify the source:

Http://blog.csdn.net/sunbow0

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Spark1.4.1 Compilation and Installation