Deploy an ApacheSpark cluster in Ubuntu

Source: Internet
Author: User
Tags install mongodb mesosphere apache mesos
1. software environment This article describes how to deploy ApacheSpark StandaloneCluster on Ubuntu. The required software is as follows: Ubuntu15.10x64ApacheSpark1.5.12. everything required for installation # sudoapt-getinstallgit-y # sudoapt-add-repositoryppa: webupd8 1. Software Environment

This article describes how to deploy an Apache Spark Standalone Cluster on Ubuntu. The required software is as follows:

  • Ubuntu 15.10x64
  • Apache Spark 1.5.1
2. everything required for Installation
# sudo apt-get install git -y# sudo apt-add-repository ppa:webupd8team/java -y# sudo apt-get update -y# sudo apt-get install Oracle-java8-installer -y# sudo apt-get install oracle-java8-set-default # sudo apt-get install maven gradle -y# sudo apt-get install sbt -y# sudo wget http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-Hadoop2.6.tgz# sudo tar -xvf spark*.tgz# sudo chmod 755 spark*# sudo apt-get update# sudo apt-get install -y openjdk-7-jdk# sudo apt-get install -y autoconf libtool# sudo apt-get -y install build-essential python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev# sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BFDISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]')CODENAME=$(lsb_release -cs)

Add to software Repository:

# echo "deb http://repos.mesosphere.io/${DISTRO} ${CODENAME} main" | \ sudo tee /etc/apt/sources.list.d/mesosphere.list# sudo apt-get -y update# sudo apt-get -y install mesos

Apache Mesos is also installed to facilitate the upgrade of the Spark cluster from the independent cluster mode in the future.

Spark-1.5.1-bin-hadoop2.6 is used for Spark standalone Clusters

conf/spark-env.sh#!/usr/bin/env bashexport SPARK_LOCAL_IP=MYIP
3. Start a node
# sbin/start-slave.sh masterIP:7077

For more information, see:

  • Http://spark.apache.org/docs/latest/running-on-mesos.html
  • Https://mesosphere.com/downloads/
  • Https://spark.apache.org/downloads.html
4. install other tools and servers

1) install MongoDB 3.0.4

# sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10# echo "deb http://repo.mongodb.org/apt/ubuntu "$(lsb_release -sc)"/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list# sudo apt-get update# sudo apt-get install -y mongodb-org# sudo apt-get install -y mongodb-org=3.0.4 mongodb-org-server=3.0.4 mongodb-org-shell=3.0.4 mongodb-org-mongos=3.0.4 mongodb-org-tools=3.0.4# sudo service mongod start# sudo tail -5000 /var/log/mongodb/mongod.log

2) install PostgreSQL

For more information, see:
Https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-14-04

# sudo apt-get update# sudo apt-get install postgresql postgresql-contrib

3) install Redis

For more information, see:
Https://www.digitalocean.com/community/tutorials/how-to-install-and-use-redis

# sudo apt-get install build-essential# sudo apt-get install tcl8.5# sudo wget http://download.redis.io/releases/redis-stable.tar.gz# sudo tar xzf redis-stable.tar.gz# cd redis-stable# make# make test# sudo make install# cd utils# sudo ./install_server.sh# sudo service redis_6379 start# redis-cli

4) install Scala 2.11.7

For more information, see:

  • Http://blog.prabeeshk.com/blog/2014/10/31/install-apache-spark-on-ubuntu-14-dot-04/
  • Http://www.scala-lang.org/download/2.11.7.html

Run the following command:

# sudo wget http://downloads.typesafe.com/scala/2.11.7/scala-2.11.7.deb# sudo dpkg -i scala-2.11.7.deb

For more information, see:
Http://www.scala-sbt.org/0.13/tutorial/Installing-sbt-on-Linux.html

# echo "deb http://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list# sudo apt-get update# sudo apt-get install sbt# sudo apt-get install unzip# curl -s get.gvmtool.net | bash# source "/root/.gvm/bin/gvm-init.sh"# gvm install gradle

For more Spark tutorials, see the following content::

Install and configure the Spark http://www.linuxidc.com/Linux/2015-08/122284.htm in CentOS 7.0

Spark1.0.0 Deployment GuideHttp://www.linuxidc.com/Linux/2014-07/104304.htm

Install Spark0.8.0 detail record http://www.linuxidc.com/Linux/2014-06/102583.htm under CentOS 6.2 (64-bit)

Introduction to Spark and its installation and use http://www.linuxidc.com/Linux/2013-08/88606.htm in Ubuntu

Install the Spark cluster (on CentOS) http://www.linuxidc.com/Linux/2013-08/88599.htm

Hadoop vs Spark performance comparison http://www.linuxidc.com/Linux/2013-08/88597.htm

Spark setup and learning http://www.linuxidc.com/Linux/2013-08/88596.htm

Spark concurrent computing model http://www.linuxidc.com/Linux/2012-12/76490.htm

Spark details: Click here
Spark: Click here

This article permanently updates the link address: Http://www.linuxidc.com/Linux/2016-01/127002.htm

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.