1, download Spark source extract to directory/usr/local/spark-1.5.0-cdh5.5.1, see if there is pom.xml file 2, switch to directory/usr/local/spark-1.5.0-cdh5.5.1 execution: When compiling the spark source code, you need to download the dependency package from the Internet, so the entire compilation process machine must be in the networked state. The compilation executes the following script:
[hadoop@hadoop spark-1.5.0-cdh5.5.1]$ export maven_opts= "-xmx2g -xx:maxpermsize=512m -x:reservedcodecachesize=512m "[hadoop@hadoop spark-1.5.0-cdh5.5.1] $mvn-pyarn-dhadoop.version=2.5.0- cdh5.3.3 -dscala-3.9.9-phive-phive-thriftserver -dskiptests Clean Package compilation error Resolution: vi /usr/local/ Maven/conf/settings.xml Add red section: <servers> <server> <id>hadoop</id> <username>hadoop</username > <password>hadoop</password> <privateKey>/home/hadoop/.ssh/id_rsa</privateKey> < Passphrase>some_passphrase</passphrase> <filepermissions>664 </filePermissions> <directorypermissions>775</directorypErmissions> <configuration></configuration> </server> </servers> vi /usr/local/spark/pom.xml Add the following red section: <modules > <module>core</module> <module>bagel</ Module> <module>graphx</module> <module> Mllib</module> <module>tools</module> < Module>network/common</module> <module>network/shuffle</module> <module>streaming</module> <module>sql/ Catalyst</module> <module>sql/core</module> <module>sql/hive</module> <module>unsafe</module> <moDule>assembly</module> <module>external/twitter</module> <module>external/flume</module> <module>external/ Flume-sink</module> <!--&NBSP;DISABLED&NBSP;IN&NBSP;CDH. <module>external/flume-assembly</module> --> <module>sql/hive-thriftserver</module> <module >external/mqtt</module> <module>external/mqtt-assembly</module> <module>external/zeromq</module> <module> Examples</module> <module>repl</module> < Module>launcher</module> <module>external/kafka</module> <!--&NBSP;DISABLED&NBSP;IN&NBSP;CDH <module>external/kafka-assembly</module> --> </modules> 3, Build Spark Deployment package:./make-distribution.sh --name 2.5.0 --tgz --with-tachyon
There is a script make-distribution.sh that generates the deployment package under the Spark source root directory, which can be packaged by executing the following command./make-distribution.sh [--name] [--tgz] [-- With-tachyon] <maven Build options>
L --name name and--tgz combine to generate spark-$VERSION-bin-$NAME. TGZ deployment package, when this parameter is not added, name is the version number of Hadoop
L --tgz generate spark-$VERSION-bin.tgz in root directory, do not generate tgz file without this parameter, only generate/dist directory
L --with-tachyon supports memory file system Tachyon, does not support Tachyon when this parameter is not added
