Spark + Hadoop-2.2.0 Environment construction under pseudo-distribution environment

The last time I introduced the installation of Spark in Hadoop mode, we will introduce the build of the spark environment based on the Hadoop pseudo-distribution mode, where Hadoop is the hadoop-2.2.0 environment and the system is ubuntu-14.04

1. First make sure that Spark has successfully unzipped the installation, see the previous blog post for details
http://blog.csdn.net/qiumianderen5/article/details/46400885

2. Then modify the spark-env.sh file inside the Conf in the Spark directory
(1) First you need to create a new spark-env.sh file from the Spark-env.sh.template

CP Spark-env.sh.template spark-env.sh

(2) then modify the parameters inside the spark-env.sh

Vim spark-env.sh

What to modify:

Export scala_home=/usr/local/share/scala/scala-2.10.4 export
java_home=/usr/jdk1.7.0_51
export Spark_ Master=localhost
export spark_local_ip=localhost
export hadoop_home=/home/hadoop/hadoop/hadoop-2.2.0
Export spark_home=/home/hadoop/hadoop/spark-1.3.1
export spark_libary_path=.: $JAVA _home/lib: $JAVA _home/ Jre/lib: $HADOOP _home/lib/native
export yarn_conf_dir= $HADOOP _home/etc/hadoop
export spark_example_jar=/ Home/hadoop/hadoop/spark-1.3.1/examples/target/spark-examples_2.10-1.3.1.jar

3. Then start Spark:

sbin/start-all.sh

4. If you need to start HDFs or yarn, you need to go to Hadoop to start the appropriate HDFs and yarn

To start HDFs:

start-dfs.sh

Start yarn:

start-yarn.sh

5. Effects

The master and worker are the spark's initiating process.

I hope we can all build a successful