Installation of "Hadoop" Spark2.0.2 on Hadoop2.7.3

1. Install Scala

A download Address: http://www.scala-lang.org/download/
I choose to install the latest version of Scala-2.12.0.tgz.

b upload the compression to the/usr/local directory

C Decompression TAR-ZXVF scala-2.12.0.tgz

D Change Soft connection
Ln-s scala-2.12.0 Scala

E Modifying configuration file Installation
Vim/etc/profile
#add by Lekko
Export Scala_home=/usr/local/scala
Export Path= Path:path:scala_home/bin

F After the configuration is complete, let it take effect
Source/etc/profile

G to see if the installed Scala version number is OK and can be executed to indicate that it has been successfully installed
Scala-version

2. Spark Installation and configuration
A download: http://spark.apache.org/downloads.html
Select the latest version 2.02

b upload the compression to the/usr/local directory

C Decompression TAR-ZXVF spark-2.0.2-bin-hadoop2.7.tgz

D Change Soft connection
Ln-s spark-2.0.2-bin-hadoop2.7 Spark

E Modifying configuration file Installation
Vim/etc/profile
#add by Lekko
Export Spark_home=/usr/local/spark
Export path= path:path:spark_home/bin: $SPARK _home/sbin

F After the configuration is complete, let it take effect
Source/etc/profile

G test environment variable setting is OK, can execute indicates successful installation
Spark-shell–version

H Configuring Spark
Modify Spark-env.sh
cd/usr/local/spark/conf/
CP Spark-env.sh.template spark-env.sh
Vim spark-env.sh
#追加如下内容
Export Scala_home=/usr/local/scala
Export Java_home=/usr/local/jdkaddress/xxxx
Export Spark_master_ip=192.168.xxx.xxx
Export spark_worker_memory=1024m
Export Hadoop_conf_dir=/usr/local/hadoop/etc/hadoop

I related start Stop command
Start Spark
start-all.sh
Recommended Use
Start-dfs.sh and start-yarn.sh
Stop command
stop-all.sh
Recommended Use
Stop-dfs.sh and stop-yarn.sh

If you see:

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh starting namenodes on [master] master:starting Namenode, logging To/home/hado Op/hadoop-2.7.3/logs/hadoop-root-namenode-master.out 121.199.7.226:starting Datanode, logging to/home/hadoop/ Hadoop-2.7.3/logs/hadoop-root-datanode-slave1.out 121.199.51.129:starting Datanode, logging to/home/hadoop/ Hadoop-2.7.3/logs/hadoop-root-datanode-slave2.out 121.199.51.174:starting Datanode, logging to/home/hadoop/ Hadoop-2.7.3/logs/hadoop-root-datanode-slave3.out starting secondary namenodes [master] master:starting Secondarynamenode, logging to/home/hadoop/hadoop-2.7.3/logs/hadoop-root-secondarynamenode-master.out starting yarn
Daemons starting ResourceManager, logging to/home/hadoop/hadoop-2.7.3/logs/yarn-root-resourcemanager-master.out
121.199.51.129:starting NodeManager, logging to/home/hadoop/hadoop-2.7.3/logs/yarn-root-nodemanager-slave2.out 121.199.7.226:starting NodeManager, logging to/home/hadoop/hadoop-2.7.3/lOgs/yarn-root-nodemanager-slave1.out 121.199.51.174:starting NodeManager, logging to/home/hadoop/hadoop-2.7.3/logs /yarn-root-nodemanager-slave3.out

Can be known to mean starting Hadoop and spark success

J Submit task to spark cluster
Spark-submit–master Spark://192.xxx.xxx.xxx:7077–class Main Function Entry –name name the full path of the jar package
Example: Spark-submit–master spark://192.xxx.xxx.xxx:7077–class CN. Xxxx. Xxxxxxxxx. Tfidf–name XXXX Xxxx.jar

K Submit task to yarn
Spark-submit–master Yarn-cluster–class CN. Xxxx. Xxxxxxxxx. Tfidf–name XXXX Xxxx.jar

View status by http://192.XXX.XXX.XXX:8088/