The last time I introduced the installation of Spark in Hadoop mode, we will introduce the build of the spark environment based on the Hadoop pseudo-distribution mode, where Hadoop is the hadoop-2.2.0 environment and the system is ubuntu-14.04
1. First make sure that Spark has successfully unzipped the installation, see the previous blog post for details
http://blog.csdn.net/qiumianderen5/article/details/46400885
2. Then modify the spark-env.sh file inside the Conf in the Spark directory
(1) First you need to create a new spark-env.sh file from the Spark-env.sh.template
CP Spark-env.sh.template spark-env.sh
(2) then modify the parameters inside the spark-env.sh
Vim spark-env.sh
What to modify:
Export scala_home=/usr/local/share/scala/scala-2.10.4 export
java_home=/usr/jdk1.7.0_51
export Spark_ Master=localhost
export spark_local_ip=localhost
export hadoop_home=/home/hadoop/hadoop/hadoop-2.2.0
Export spark_home=/home/hadoop/hadoop/spark-1.3.1
export spark_libary_path=.: $JAVA _home/lib: $JAVA _home/ Jre/lib: $HADOOP _home/lib/native
export yarn_conf_dir= $HADOOP _home/etc/hadoop
export spark_example_jar=/ Home/hadoop/hadoop/spark-1.3.1/examples/target/spark-examples_2.10-1.3.1.jar
3. Then start Spark:
sbin/start-all.sh
4. If you need to start HDFs or yarn, you need to go to Hadoop to start the appropriate HDFs and yarn
To start HDFs:
start-dfs.sh
Start yarn:
start-yarn.sh
5. Effects
The master and worker are the spark's initiating process.
I hope we can all build a successful