CentOS 7 installed Spark 2.2 stand-alone

Spark 2.2 needs to run in the Java environment, is not any version can do. Not Before the spark official website to see the need to run the environment is jdk8+, so I installed Jdk9, but in the JDK 9 environment did not install successfully, run Spark./bin/spark-shell will report a loss of configuration value (estimated as a JDK version of the problem). During this installation, JDK 8 has been selected and is running normally after installation. The following simple record of the installation process, relatively simple.

1. Download jdk,http://www.oracle.com/technetwork/java/javase/downloads/index.html, select JDK 8 download

2. Install JDK 8. RPM-IVH jdk-8u151-linux-x64.rpm

3. Set Java_home. At the end of the/etc/profile, add the following two lines

Export java_home=/usr/java/jdk1.8.0_151
export path= $PATH: $JAVA _home/bin

Set up Java home according to the actual directory you have installed.

4. Download Spark. Http://spark.apache.org/downloads.html

Select 2.2.0 version, package type select default "pre-built for Apache Hadoop 2.7 and later"

5. Copy to the directory to be installed, decompression

Tar zxvf spark-2.2.0-bin-hadoop2.7.tgz

6. Run Spark

./sbin/start-master.sh

Check the logs in the logs for errors

7. Run Spark-shell

./bin/spark-shell

If there is no mistake, the installation is successful.

In addition, if you are using Python, you can install Python and then run the./bin/pyspark. I was successfully installed on the Python 3.6.3. Here's how to install Python 3.6 simple

8. Download Python 3.6

Https://www.python.org/ftp/python/3.6.3/Python-3.6.3.tar.xz

9. Install XZ Decompression Tool

Yum Install XZ

10. Unpack the installation package

xz-d python-3.6.3.tar.xz
Tar xvf python-3.6.3.tar

11. Compile and install to see the blog of Not afraid brother Http://www.cnblogs.com/cloud-80808174-sea/p/6902934.html

One need to note is that after the spark version 2.10 no longer support Python2.7, so after the installation Python3.6.3, you need to set the 3.6.3 as the default, in the not Afraid Brother's blog has written how to change.