Spark 1.0.0 is deployed on hadoop 2.2.0.

Source code compilation

 

My test environment:

 

• System: centos 6.4-64-bit
• Java: 1.7.45
• Scala: 2.10.4
• Hadoop: 2.2.0

 

Spark 1.0.0 source code: http://d3kbcqa49mib13.cloudfront.net/spark-1.0.0.tgz

Decompress the source code and execute the following command in the root directory (SBT compilation is not attempted)

./Make-distribution.sh -- hadoop 2.2.0 -- With-yarn -- tgz -- With-hive

Several important parameters

-- Hadoop: Specifies the hadoop version.

-- With-yarn support is required

-- With-hive is also required to read hive data. Anyway, I hate shark. In the future, developers can encapsulate SQL & hql clients on spark, which is also a good choice.

# -- Tgz: Additionally creates spark-1_version-bin.tar.gz
# -- Hadoop version: builds against specified version of hadoop.
# -- With-yarn: Enables support for hadoop yarn.
# -- With-Hive: enable support for reading hive tables.
# -- Name: a moniker for the release target. defaults to the hadoop verison.

Download the Binary Package directly if you don't want to compile it yourself:

Spark 1.0.0 on hadoop 1/cdh3, cdh4 Binary Package: http://d3kbcqa49mib13.cloudfront.net/spark-1.0.0-bin-hadoop1.tgz

Spark 1.0.0 on hadoop 2/cdh5, hdp2 Binary Package: http://d3kbcqa49mib13.cloudfront.net/spark-1.0.0-bin-hadoop2.tgz

After a long wait, a tgz compressed package will be generated under the source code and directory.

 

 

 

Copy the package to the directory you want to deploy and decompress it.

 

Environment variable:

 

Export scala_home =/opt/scala-2.10.4

Export Path = $ path: $ scala_home/bin

Export spark_home =/opt/spark

Export Path = $ path: $ spark_home/bin: $ spark_home/sbin

 

 

Note: you only need to copy the decompressed package to any of the yarn clusters. One node is enough and does not need to be deployed on all nodes unless you need multiple client nodes to call spark jobs.

Here, we do not need to build an independent spark cluster and use yarn client to call the computing resources of the hadoop cluster.

 

[Plain]View plaincopy

1. Directory/CONF/log4j. properties. template after CP decompression/CONF/log4j. Properties
2. CP decompressed directory/CONF/spark-env.sh.template decompressed directory/CONF/spark-env.sh

Edit spark-env.sh

 

 

 

Export hadoop_conf_dir =/opt/hadoop/etc/hadoop

Export spark_master_ip = Master

Export spark _ master_port = 7077

Export scala_home =/opt/scala-2.10.4

Export java_home =/opt/jdk1.7.0 _ 45

 

Edit history Server

You can view the history of completed jobs by configuring the conf/spark-defaults.conf

Spark. EventLog. Enabled = true

Spark. EventLog. dir = HDFS: // xx

For detailed configuration, see spark1.0.0 history server configuration.

 

This is my configuration. The configuration is slightly different from the previous versions, but the big difference is not bad.

 

Use yarn client to call the classic Mr example: Word Count of spark

Note that sparkcontext is changed. The first parameter in the wordcount example of the previous version should be removed.

 

[Plain]View plaincopy

1. Spark_jar =./lib/spark-assembly-1.0.0-hadoop2.2.0.jar \
2. ./Bin/spark-class org. Apache. Spark. Deploy. yarn. Client \
3. -- Jar./lib/spark-examples-1.0.0-hadoop2.2.0.jar \
4. -- Class org. Apache. Spark. Examples. javawordcount \
5. -- ARGs HDFS: // master: 9000/user/hadoop/readme. md \
6. -- Num-executors 2 \
7. -- Executor-cores 1 \
8. -- Driver-memory 1024 m \
9. -- Executor-memory 1000 m \
10. -- Name "word count on spark"

View the running result in stdout