Build a spark development environment on your Mac

This article describes building a spark development environment on Mac OS.

Before installing the spark environment, you need to ensure that the appropriate Java environment is configured, and that the JDK or JRE version needs to be in the 1.8.0 version.

(1) First download the development IDE, the current more popular IDE is IntelliJ idea, can be downloaded in the official website,: https://www.jetbrains.com/idea/.

(2) Spark Environment setup:
Start the terminal on your Mac, use the Brew Install Scala command (brew is a package management tool similar to the one under CentOS or Apt-get under Ubuntu) and download the Scala installation as follows:

This Scala environment has been successfully installed, and now we download spark-related environments from Spark's website at: http://spark.apache.org/

After downloading Spark, go to the folder of the downloaded spark package and execute TAR-ZXVF spark-2.3.1-bin-hadoop2.7.tar.

After installing Scala and spark, configure the relevant environment variables, execute the vim/etc/profile command, and then append the following environment variables:

Export scala_home=/usr/local/cellar/scala/2.12.4
Export path= $PATH: $SCALA _home/bin
Export spark_home=/users/mengxin/downloads/spark-2.3.1-bin-hadoop2.7
Export path= $PATH: $SPARK _home/bin

(3) for the native SSH configuration, the home directory executes the following command:

Ssh-keygen-t rsa-p ""

Cat ~/.ssh/id_rsa.pub>>~/.ssh/authorized_keys

The execution process is as follows:

(4) Detect if Spark can start

Enter the Sbin directory under the Spark decompression package to execute the./start-all.sh, as

You can view the relevant startup conditions under the Local.out file.

When Spark starts, the master and worker processes that are started are actually JVM processes, which we can look at with the JPS command.

Then, with the spark shell Test, go to the bin directory of the spark installation package, and if the following interface appears, the Spark installation environment is done.

Build a spark development environment on your Mac