Build a Spark development environment in Ubuntu

Unzip, decompress it to/opt/jdk1.8.0 _ 45: http://www.oracle.com/technetwork/java/javase/downloads/index.htmlinstall Scala, download scala-2.11.6.tg

Configure Ubuntu to use Python to develop Spark applications

Ubuntu 64-bit basic environment Configuration

Install JDK, download jdk-8u45-linux-x64.tar.gz, and decompress it to/opt/jdk1.8.0 _ 45.

: Http://www.Oracle.com/technetwork/java/javase/downloads/index.html

Install scala, download scala-2.11.6.tgz, unzip to/opt/scala-2.11.6

Address: http://www.scala-lang.org/

Install Spark, download spark-1.3.1-bin-Hadoop2.6.tgz, unzip to/opt/spark-hadoop

: Http://spark.apache.org/downloads.html,

Configure environment variables, edit/etc/profile, and execute the following command

Python @ ubuntu :~ $ Sudo gedit/etc/profile

Add at most files:

# Seeting JDK Environment Variables

Export JAVA_HOME =/opt/jdk1.8.0 _ 45

Export JRE_HOME =$ {JAVA_HOME}/jre

Export CLASSPATH =. :$ {JAVA_HOME}/lib :$ {JRE_HOME}/lib

Export PATH =$ {JAVA_HOME}/bin :$ {JRE_HOME}/bin: $ PATH

# Seeting Scala environment variable

Export SCALA_HOME =/opt/scala-2.11.6

Export PATH =$ {SCALA_HOME}/bin: $ PATH

# Setting Spark environment variable

Export SPARK_HOME =/opt/spark-hadoop/

# PythonPath: add the Python Environment added to the pySpark module in Spark

Export PYTHONPATH =/opt/spark-hadoop/python

Restart the computer to make the/etc/profile take effect permanently and take effect temporarily. Open the command window and execute source/etc/profile to take effect in the current window.

• Test installation result

• Open the command window and switch to the Spark root directory.

• Run./bin/spark-shell to open the connection window from Scala to Spark.

No error message during startup. scala> is displayed. the startup is successful.

• Run./bin/pyspark to open the connection window from Python to Spark.

There is no error during the startup process. If the preceding error occurs, the startup is successful.

• Access through a browser: The following page appears

Test SPark availability.

• Python amfa Spark Application

• You have set PYTHONPATH to add pyspark to the search path of Python.

• Open the Spark installation directory, and copy py4j under the Python-build folder to the Python directory,

• Open the command line window and enter python. The Python version is 2.7.6. Note that Spark does not support Python3.

• Enter import pyspark, as shown in, to prove that the work is completed before development.

• Use Pycharm to create a project and use the code in the red box to test the project:

For more Spark tutorials, see the following content::

Install and configure the Spark http://www.linuxidc.com/Linux/2015-08/122284.htm in CentOS 7.0

Spark1.0.0 Deployment GuideHttp://www.linuxidc.com/Linux/2014-07/104304.htm

Install Spark0.8.0 detail record http://www.linuxidc.com/Linux/2014-06/102583.htm under CentOS 6.2 (64-bit)

Introduction to Spark and its installation and use http://www.linuxidc.com/Linux/2013-08/88606.htm in Ubuntu

Install the Spark cluster (on CentOS) http://www.linuxidc.com/Linux/2013-08/88599.htm

Hadoop vs Spark performance comparison http://www.linuxidc.com/Linux/2013-08/88597.htm

Spark setup and learning http://www.linuxidc.com/Linux/2013-08/88596.htm

Spark concurrent computing model http://www.linuxidc.com/Linux/2012-12/76490.htm

Spark details: Click here
Spark: Click here

This article permanently updates the link address: Http://www.linuxidc.com/Linux/2016-01/127855.htm