Compilation of Spark Source code package

Compilation of Spark Source code package

There are only two types of Spark Source code package: Maven and SBT (Simple Build Tool), which are only applicable to different scenarios:

Maven Compilation

SBT Compilation

IntelliJ IDEA compilation (can be compiled using Maven or SBT plug-in), suitable for developers

Package deployment generation (embedded Maven compilation), using maintenance personnel

The purpose of compilation is to generate a Spark JAR package for running Spark itself or developing Spark Application in a specific environment. The purpose of this compilation is to generate a Spark JAR package running on Hadoop2.5.2, the Hadoop environment supported by the default compilation is Hadoop1.0.4.

SBT Compilation

Copy the source code package to the specified directory, enter the directory, and run:

SPARK_HADOOP_VERSION = 2.5.2 SPARK_YARN = true sbt/sbt assembly

Maven Compilation

Install Maven3.3.2 or Maven3.0.5 in advance, set the environment variable MAVEN_HOME, and add MAVEN_HOME/bin to the PATH variable. Copy the source code to the specified directory and enter the directory. Set the Maven parameter first.

Export MAVEN_OPTS = "-Xmx2g-XX: MaxPermSize = 512 M-XX: ReservedCodeCacheSize = 512 m"

Run again

Mvn-Pyarn-Dhadoop. version = 2.5.2-Dyarn. version = 2.5.2-DskipTests clean package

-------------------------------------- Split line --------------------------------------

Spark1.0.0 Deployment Guide

Install Spark0.8.0 in CentOS 6.2 (64-bit)

Introduction to Spark and its installation and use in Ubuntu

Install the Spark cluster (on CentOS)

Hadoop vs Spark Performance Comparison

Spark installation and learning

Spark Parallel Computing Model

-------------------------------------- Split line --------------------------------------

Spark details: click here
Spark: click here

This article permanently updates the link address: