Compiling spark Source code

                                            This example records the process and problems of Spark source code compilation


Because the compilation will have a lot of inexplicable errors, for convenience, using the CDH version of Hadoop, note that the version is consistent with mine,
Environment: maven3.0.5        scala2.10.4                            :Http://www.scala-lang.org/download/all.htmlspark-1.3.0-src                    :Http://spark.apache.org/downloads.htmlhadoop version: hadoop-2.6.0-cdh5.4.0.tar.gz                : http://archive.cloudera.com/cdh5/cdh/5/                Size: 282M
How to: make-distribution.sh Packaging (with maven)
Command: Enter the SPARK directory CD $SPARK _home
./make-distribution.sh--tgz-pyarn-phadoop-2.4-dhadoop.version=2.6.0-cdh5.4.0-phive-0.13.1-phive-thriftserver


--------------------------------------------------------------------------------------------------------------- ---------------------, pre-compilation 1, compile the image configuration because my   /root/.m2 Directory does not have settings files, so set the global, in the Conf directory, you can CD $MAVEN _home/.m2, make Settings        the settings file in the Maven conf directory joins:

<mirror><id>nexus-osc</id><mirrorof>*</mirrorof><name>nexus Osc</name ><url>http://maven.oschina.net/content/groups/public/</url></mirror>       
2, Domain Configurationmaven compiled easily by the wall, so add the following two domain names, it is not easy to error
vi/etc/resolv.conf                        Add two domains
                            

3,
Because the make-distribution.sh execution time will find the version of the action itself, in order to speed up, the following code commented out, manually add the version information,
The following changes are-------in make-distribution.sh

Explain:
version=1.3.0                    version of//sparkspark_hadoop_version=2.6.0-cdh5.4.0    version of//hadoopSpark_hive=1                            //hive,    1 indicates that hive needs to be packaged in, not 1 digits to not pack hive


a spark-1.3.0-bin-2.6.0-cdh5.4.0.tgz is generated after the compilation is completed



This is the cloud sail Big Data teacher gives the method I sort out, can also directly use the official given the compiled jar package, such as given the build-hadoop2.6, then download the corresponding version of Hadoop use


























From for notes (Wiz)

Compiling spark Source code