Deployment and development environment construction of Apache Spark1.1.0

Deployment and development environment construction of Apache Spark1.1.0

Spark is a parallel computing architecture launched by Apache based on Hadoop Distributed File System (HDFS. Unlike MapReduce, Spark is not limited to writing map and reduce methods. It provides a more powerful in-memory computing model, this allows you to program and read data to the memory of the cluster, and allows you to quickly and repeatedly query data. It is very suitable for implementing machine learning algorithms. This article describes how to deploy and build an Apache Spark1.1.0 development environment.

For more Spark tutorials, see the following:

Install and configure Spark in CentOS 7.0

Spark1.0.0 Deployment Guide

Install Spark0.8.0 in CentOS 6.2 (64-bit)

Introduction to Spark and its installation and use in Ubuntu

Install the Spark cluster (on CentOS)

Hadoop vs Spark Performance Comparison

Spark installation and learning

Spark Parallel Computing Model

0. Prepare

For the purpose of learning, this article deploys Spark in a virtual machine and selects VMware WorkStation as the virtual machine. Install the following software on a virtual machine:

• Ubuntu 14.04.1 LTS 64-bit Desktop
• Hadoop-2.4.0.tar.gz
• Jdk-7u67-linux-x64.tar.gz
• Scala-2.10.4.tgz
• Spark-1.1.0-bin-hadoop2.4.tgz

Spark development environment. This article selects Windows7 platform and IDE IntelliJ IDEA. Install the following software in Windows:

• IntelliJ IDEA 13.1.4 Community Edition
• Apache-maven-3.2.3-bin.zip (installation process is relatively simple, please install your own)

1. Install JDK

Decompress the jdk installation package to the/usr/lib directory:

1 sudo cp jdk-7u67-linux-x64.gz /usr/lib2 cd /usr/lib3 sudo tar -xvzf jdk-7u67-linux-x64.gz4 sudo gedit /etc/profile

Add the environment variable at the end of the/etc/profile file:

1 export JAVA_HOME=/usr/lib/jdk1.7.0_672 export JRE_HOME=/usr/lib/jdk1.7.0_67/jre3 export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH4 export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH

Save and update/etc/profile:

1 source /etc/profile

Test whether jdk is successfully installed:

1 java -version

 

2. install and configure SSH

1 sudo apt-get update2 sudo apt-get install openssh-server3 sudo /etc/init.d/ssh start

Generate and add a key:

1 ssh-keygen -t rsa -P ""  2 cd /home/hduser/.ssh 3 cat id_rsa.pub >> authorized_keys

Ssh Logon:

1 ssh localhost

 

3. Install hadoop2.4.0

Install hadoop2.4.0 In pseudo distribution mode. Decompress hadoop2.4.0 to the/usr/local directory:

1 sudo cp hadoop-2.4.0.tar.gz /usr/local/2 sudo tar -xzvf hadoop-2.4.0.tar.gz

Add the environment variable at the end of the/etc/profile file:

1 export HADOOP_HOME=/usr/local/hadoop-2.4.02 export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH3 4 export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native5 export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

Save and update/etc/profile:

1 source /etc/profile

Modify the jdk path in the hadoop-2.4.0 and hadoop-env.sh file at/usr/local/yarn-env.sh/etc/hadoop:

1 cd /usr/local/hadoop-2.4.0/etc/hadoop2 sudo gedit hadoop-env.sh3 sudo gedit yarn-evn.sh

Hadoop-env.sh:

Yarn-env.sh:

Modify core-site.xml:

1 sudo gedit core-site.xml

Add:

1 <property>2   <name>fs.default.name</name>3   <value>hdfs://localhost:9000</value>4 </property>5 6 <property>7   <name>hadoop.tmp.dir</name>8   <value>/app/hadoop/tmp</value>9 </property>

Modify hdfs-site.xml:

1 sudo gedit hdfs-site.xml

Add:

1 <property>
2 <name> dfs. namenode. name. dir </name>
3 <value>/app/hadoop/dfs/nn </value>
4 </property>
5
6 <property>
7 <name> dfs. namenode. data. dir </name>
8 <value>/app/hadoop/dfs/dn </value>
9 </property>
10
11 <property>
12 <name> dfs. replication </name>
13 <value> 1 </value>
14 </property>

Modify yarn-site.xml:

1 sudo gedit yarn-site.xml

Add:

1 <property>2   <name>mapreduce.framework.name</name>3   <value>yarn</value>4 </property>5 6 <property>7   <name>yarn.nodemanager.aux-services</name>8   <value>mapreduce_shuffle</value>9 </property>

Copy and rename mapred-site.xml.template to mapred-site.xml:

1 sudo cp mapred-site.xml.template mapred-site.xml2 sudo gedit mapred-site.xml

Add:

1 <property>2   <name>mapreduce.jobtracker.address </name>3   <value>hdfs://localhost:9001</value>4 </property>

Before starting hadoop, remember to set permissions for the/app directory to prevent possible log writing failures:

1 sudo mkdir /app2 sudo chmod -R hduser:hduser /app

Format hadoop:

1 hadoop namenode -format 

Start hdfs and yarn. When developing Spark, you only need to start hdfs:

1 sbin/start-dfs.sh  2 sbin/start-yarn.sh  

Open the address http: // localhost: 50070/in the browser to view the hdfs status information:

 

4. Install scala

1 sudo cp /home/hduser/Download/scala-2.9.3.tgz /usr/local2 sudo tar -xvzf scala-2.9.3.tgz 

Add the environment variable at the end of the/etc/profile file:

1 export SCALA_HOME=/usr/local/scala-2.9.32 export PATH=$SCALA_HOME/bin:$PATH

Save and update/etc/profile:

1 source /etc/profile

Test whether scala is successfully installed:

1 scala -version

 

5. Install Spark

1 sudo cp spark-1.1.0-bin-hadoop2.4.tgz /usr/local2 sudo tar -xvzf spark-1.1.0-bin-hadoop2.4.tgz 

Add the environment variable at the end of the/etc/profile file:

1 export SPARK_HOME=/usr/local/spark-1.1.0-bin-hadoop2.42 export PATH=$SPARK_HOME/bin:$PATH

Save and update/etc/profile:

1 source /etc/profile

Copy and rename spark-env.sh.template to spark-env.sh:

1 sudo cp spark-env.sh.template spark-env.sh2 sudo gedit spark-env.sh

Add in spark-env.sh:

1 export SCALA_HOME=/usr/local/scala-2.9.32 export JAVA_HOME=/usr/lib/jdk1.7.0_673 export SPARK_MASTER_IP=localhost4 export SPARK_WORKER_MEMORY=1000m

Start Spark:

1 cd /usr/local/spark-1.1.0-bin-hadoop2.42 sbin/start-all.sh

Test whether Spark is successfully installed:

1 cd /usr/local/spark-1.1.0-bin-hadoop2.42 bin/run-example SparkPi

 

6. Build a Spark Development Environment

IntelliJ IDEA is recommended for Spark IDE development in this article. You can also choose Eclipse. Install the scala plug-in before using IntelliJ IDEA. Click Configure:

Click Plugins:

Click Browse repositories ...:

Enter scala in the search box and select Scala plug-in for installation. The Installation option is not displayed because the plug-in has been installed:

After the installation is complete, IntelliJ IDEA requires a restart.

For more details, please continue to read the highlights on the next page:

1 2 Next Page