The following information is included:http://blog.csdn.net/lovehuangjiaju/article/details/484947371. Extracting installation files, setting environment variables the installation files we use here are already compiled spark, Need to be paired with hadoop-2.6.0. The download link is: HTTP://PAN.BAIDU.COM/S/1GDNW5MR unzip the installation file to the/cloud directory set the environment variables for SPARK: Export Spark_home=/cloud/spark-1.5.1export PATH = $PATH: $SPARK _home/bin: $SPARK _home/sbin2. Settings related profiles (1) SPARK-ENV.SHCD/CLOUD/SPARK-1.5.1/CONFCP Spark-env.sh.template Spark-env.shvi spark-env.sh Add the following start-master.sh content to the file: Export java_home=/usr/lib/jvm/ Java-1.7.0-openjdk.x86_64export Scala_home=/cloud/scala-2.10.6export hadoop_conf_dir=/cloud/hadoop-2.6.0/etc/ Hadoop (2) SLAVESCP slaves.template Slaves
The slaves file indicates which hosts the worker is running onAdd the following to the file masterslave1slave2 that the worker will run on Master,slave1,slave2. (3) Since Storm Web-ui has occupied 8080 ports and needs to change the spark port, here we change to 8888CD. /sbinvi start-master.sh modified port number as follows: if ["$SPARK _master_webui_port" = ""]; Then spark_master_webui_port=
8888Fi3. Distributing installation files Cd/cloudscp-r spark-1.5.1/[email Protected]:/cloudscp-r spark-1.5.1/[email protected]:/cloud4. Start the Spark cluster (note to start the Hadoop cluster first)Cd/cloud/spark-1.5.1/sbin./start-all.shMaster Process Condition:Slave1 and slave2 process conditions:5. Test whether the cluster is working properly(1) Access to spark master WEB_UIhttp://master:8888
(2) running Spark version WordCount①First, the introduction file for Spark is submitted to Hdfscd/cloud/spark-1.5.1hdfs Dfs-put Readme.md② start Spark-shellCD Bin./spark-shell② statistics The number of words spark in the text line in the Readme.md file in HDFs val textcount = Sc.textfile ("readme.md"). Filter (line = Line.contains ("Spark")). Count () is visible, Spark has been installed successfully and is running correctly.
From for notes (Wiz)
Installing spark-1.5.1 in the CentOS 6.5 x64
