Build a single-host cluster of Spark

Build a single-host cluster of Spark

Build a single-host cluster of Spark

1. Create a user

# Useradd spark

# Passwd spark

Ii. Download Software

JDK, Scala, SBT, Maven

The version information is as follows:

JDK jdk-7u79-linux-x64.gz.

Scala scala-2.10.5.tgz

Sbt-0.13.7.zip SBT

Maven apache-maven-3.2.5-bin.tar.gz

Note: If you only need to install the Spark environment, you only need JDK and Scala. SBT and Maven are for subsequent source code compilation.

3. decompress the above file and configure Environment Variables

# Cd/usr/local/

# Tar xvf/root/jdk-7u79-linux-x64.gz

# Tar xvf/root/scala-2.10.5.tgz

# Tar xvf/root/apache-maven-3.2.5-bin.tar.gz

# Unzip/root/sbt-0.13.7.zip

Modify the environment variable configuration file

# Vim/etc/profile

export JAVA_HOME=/usr/local/jdk1.7.0_79export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport SCALA_HOME=/usr/local/scala-2.10.5export MAVEN_HOME=/usr/local/apache-maven-3.2.5export SBT_HOME=/usr/local/sbtexport PATH=$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin:$MAVEN_HOME/bin:$SBT_HOME/bin

Make the configuration file take effect

# Source/etc/profile

Test whether environment variables take effect

# Java-version

java version "1.7.0_79"Java(TM) SE Runtime Environment (build 1.7.0_79-b15)Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

# Scala-version

Scala code runner version 2.10.5 -- Copyright 2002-2013, LAMP/EPFL

# Mvn-version

Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-15T01:29:23+08:00)Maven home: /usr/local/apache-maven-3.2.5Java version: 1.7.0_79, vendor: Oracle CorporationJava home: /usr/local/jdk1.7.0_79/jreDefault locale: en_US, platform encoding: UTF-8OS name: "linux", version: "3.10.0-229.el7.x86_64", arch: "amd64", family: "unix"

# Sbt -- version

sbt launcher version 0.13.7

4. Host Name binding

[Root @ spark01 ~] # Vim/etc/hosts

192.168.244.147 spark01

5. Configure spark

Switch to the spark user

Download Hadoop and spark using the wget command

Spark-1.4.0 http://d3kbcqa49mib13.cloudfront.net/spark-1.4.0-bin-hadoop2.6.tgz

Hadoop http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz

Decompress the preceding file and configure environment variables.

Modify the configuration file of spark user environment variables

[Spark @ spark01 ~] $ Vim. bash_profile

export SPARK_HOME=$HOME/spark-1.4.0-bin-hadoop2.6export HADOOP_HOME=$HOME/hadoop-2.6.0export HADOOP_CONF_DIR=$HOME/hadoop-2.6.0/etc/hadoopexport PATH=$PATH:$SPARK_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Make the configuration file take effect

[Spark @ spark01 ~] $ Source. bash_profile

Modify spark configuration file

[Spark @ spark01 ~] $ Cd spark-1.4.0-bin-hadoop2.6/conf/

[Spark @ spark01 conf] $ cp spark-env.sh.template

[Spark @ spark01 conf] $ vim spark-env.sh

Add the following content later:

export SCALA_HOME=/usr/local/scala-2.10.5export SPARK_MASTER_IP=spark01export SPARK_WORKER_MEMORY=1500mexport JAVA_HOME=/usr/local/jdk1.7.0_79

If you have any requirements, you can set SPARK_WORKER_MEMORY to a larger value. Because the memory of my VM is 2 GB, only 1500 MB is assigned.

Configure slaves

[Spark @ spark01 conf] $ cp slaves. template

[Spark @ spark01 conf] $ vim slaves

Change localhost to spark01

Start master

[Spark @ spark01 spark-1.4.0-bin-hadoop2.6] $ sbin/start-master.sh

starting org.apache.spark.deploy.master.Master, logging to /home/spark/spark-1.4.0-bin-hadoop2.6/sbin/../logs/spark-spark-org.apache.spark.deploy.master.Master-1-spark01.out

View the output content of the preceding log

[Spark @ spark01 spark-1.4.0-bin-hadoop2.6] $ cd logs/

[Spark @ spark01 logs] $ cat spark-spark-org.apache.spark.deploy.master.Master-1-spark01.out

 

Spark Command: /usr/local/jdk1.7.0_79/bin/java -cp /home/spark/spark-1.4.0-bin-hadoop2.6/sbin/../conf/:/home/spark/spark-1.4.0-bin-hadoop2.6/lib/spark-assembly-1.4.0-hadoop2.6.0.jar:/home/spark/spark-1.4.0-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/home/spark/spark-1.4.0-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/home/spark/spark-1.4.0-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/home/spark/hadoop-2.6.0/etc/hadoop/ -Xms512m -Xmx512m -XX:MaxPermSize=128m org.apache.spark.deploy.master.Master --ip spark01 --port 7077 --webui-port 8080========================================16/01/16 15:12:30 INFO master.Master: Registered signal handlers for [TERM, HUP, INT]16/01/16 15:12:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable16/01/16 15:12:32 INFO spark.SecurityManager: Changing view acls to: spark16/01/16 15:12:32 INFO spark.SecurityManager: Changing modify acls to: spark16/01/16 15:12:32 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)16/01/16 15:12:33 INFO slf4j.Slf4jLogger: Slf4jLogger started16/01/16 15:12:33 INFO Remoting: Starting remoting16/01/16 15:12:33 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster@spark01:7077]16/01/16 15:12:33 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.16/01/16 15:12:34 INFO server.Server: jetty-8.y.z-SNAPSHOT16/01/16 15:12:34 INFO server.AbstractConnector: Started SelectChannelConnector@spark01:606616/01/16 15:12:34 INFO util.Utils: Successfully started service on port 6066.16/01/16 15:12:34 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 606616/01/16 15:12:34 INFO master.Master: Starting Spark master at spark://spark01:707716/01/16 15:12:34 INFO master.Master: Running Spark version 1.4.016/01/16 15:12:34 INFO server.Server: jetty-8.y.z-SNAPSHOT16/01/16 15:12:34 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:808016/01/16 15:12:34 INFO util.Utils: Successfully started service 'MasterUI' on port 8080.16/01/16 15:12:34 INFO ui.MasterWebUI: Started MasterWebUI at http://192.168.244.147:808016/01/16 15:12:34 INFO master.Master: I have been elected leader! New state: ALIVE

 

The log shows that the master instance is started normally.

Next let's take a look at the master's web management interface. The default port is port 8080.

Start worker

[Spark @ spark01 spark-1.4.0-bin-hadoop2.6] $ sbin/start-slaves.sh spark: // spark01: 7077

spark01: Warning: Permanently added 'spark01,192.168.244.147' (ECDSA) to the list of known hosts.spark@spark01's password:spark01: starting org.apache.spark.deploy.worker.Worker, logging to /home/spark/spark-1.4.0-bin-hadoop2.6/sbin/../logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-spark01.out

Enter the spark user password on spark01

You can use the log information to check whether the workder is properly started. Because there is too much information, it will not be posted here.

[Spark @ spark01 spark-1.4.0-bin-hadoop2.6] $ cd logs/

[Spark @ spark01 logs] $ cat spark-spark-org.apache.spark.deploy.worker.Worker-1-spark01.out

Start spark shell

[Spark @ spark01 spark-1.4.0-bin-hadoop2.6] $ bin/spark-shell -- master spark: // spark01: 7077

 

16/01/16 15:33:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable16/01/16 15:33:18 INFO spark.SecurityManager: Changing view acls to: spark16/01/16 15:33:18 INFO spark.SecurityManager: Changing modify acls to: spark16/01/16 15:33:18 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)16/01/16 15:33:18 INFO spark.HttpServer: Starting HTTP Server16/01/16 15:33:18 INFO server.Server: jetty-8.y.z-SNAPSHOT16/01/16 15:33:18 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:4230016/01/16 15:33:18 INFO util.Utils: Successfully started service 'HTTP class server' on port 42300.Welcome to      ____              __     / __/__  ___ _____/ /__    _\ \/ _ \/ _ `/ __/  '_/   /___/ .__/\_,_/_/ /_/\_\   version 1.4.0      /_/Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_79)Type in expressions to have them evaluated.Type :help for more information.16/01/16 15:33:30 INFO spark.SparkContext: Running Spark version 1.4.016/01/16 15:33:30 INFO spark.SecurityManager: Changing view acls to: spark16/01/16 15:33:30 INFO spark.SecurityManager: Changing modify acls to: spark16/01/16 15:33:30 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)16/01/16 15:33:31 INFO slf4j.Slf4jLogger: Slf4jLogger started16/01/16 15:33:31 INFO Remoting: Starting remoting16/01/16 15:33:31 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.244.147:43850]16/01/16 15:33:31 INFO util.Utils: Successfully started service 'sparkDriver' on port 43850.16/01/16 15:33:31 INFO spark.SparkEnv: Registering MapOutputTracker16/01/16 15:33:31 INFO spark.SparkEnv: Registering BlockManagerMaster16/01/16 15:33:31 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-7b7bd4bd-ff20-4e3d-a354-61a4ca7c4b2f/blockmgr-0e855210-3609-4204-b5e3-151e0c096c1516/01/16 15:33:31 INFO storage.MemoryStore: MemoryStore started with capacity 265.4 MB16/01/16 15:33:31 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-7b7bd4bd-ff20-4e3d-a354-61a4ca7c4b2f/httpd-56ac16d2-dd82-41cb-99d7-4d11ef36b42e16/01/16 15:33:31 INFO spark.HttpServer: Starting HTTP Server16/01/16 15:33:31 INFO server.Server: jetty-8.y.z-SNAPSHOT16/01/16 15:33:31 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:4763316/01/16 15:33:31 INFO util.Utils: Successfully started service 'HTTP file server' on port 47633.16/01/16 15:33:31 INFO spark.SparkEnv: Registering OutputCommitCoordinator16/01/16 15:33:31 INFO server.Server: jetty-8.y.z-SNAPSHOT16/01/16 15:33:31 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:404016/01/16 15:33:31 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.16/01/16 15:33:31 INFO ui.SparkUI: Started SparkUI at http://192.168.244.147:404016/01/16 15:33:32 INFO client.AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@spark01:7077/user/Master...16/01/16 15:33:33 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160116153332-000016/01/16 15:33:33 INFO client.AppClient$ClientActor: Executor added: app-20160116153332-0000/0 on worker-20160116152314-192.168.244.147-58914 (192.168.244.147:58914) with 2 cores16/01/16 15:33:33 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160116153332-0000/0 on hostPort 192.168.244.147:58914 with 2 cores, 512.0 MB RAM16/01/16 15:33:33 INFO client.AppClient$ClientActor: Executor updated: app-20160116153332-0000/0 is now LOADING16/01/16 15:33:33 INFO client.AppClient$ClientActor: Executor updated: app-20160116153332-0000/0 is now RUNNING16/01/16 15:33:34 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33146.16/01/16 15:33:34 INFO netty.NettyBlockTransferService: Server created on 3314616/01/16 15:33:34 INFO storage.BlockManagerMaster: Trying to register BlockManager16/01/16 15:33:34 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.244.147:33146 with 265.4 MB RAM, BlockManagerId(driver, 192.168.244.147, 33146)16/01/16 15:33:34 INFO storage.BlockManagerMaster: Registered BlockManager16/01/16 15:33:34 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.016/01/16 15:33:34 INFO repl.SparkILoop: Created spark context..Spark context available as sc.16/01/16 15:33:38 INFO hive.HiveContext: Initializing execution hive, version 0.13.116/01/16 15:33:43 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore16/01/16 15:33:43 INFO metastore.ObjectStore: ObjectStore, initialize called16/01/16 15:33:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored16/01/16 15:33:44 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored16/01/16 15:33:44 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@192.168.244.147:46741/user/Executor#-2043358626]) with ID 016/01/16 15:33:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)16/01/16 15:33:45 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.244.147:33017 with 265.4 MB RAM, BlockManagerId(0, 192.168.244.147, 33017)16/01/16 15:33:46 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)16/01/16 15:33:48 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"16/01/16 15:33:48 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5.  Encountered: "@" (64), after : "".16/01/16 15:33:52 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.16/01/16 15:33:52 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.16/01/16 15:33:54 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.16/01/16 15:33:54 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.16/01/16 15:33:54 INFO metastore.ObjectStore: Initialized ObjectStore16/01/16 15:33:54 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa16/01/16 15:33:55 INFO metastore.HiveMetaStore: Added admin role in metastore16/01/16 15:33:55 INFO metastore.HiveMetaStore: Added public role in metastore16/01/16 15:33:56 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty16/01/16 15:33:56 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.16/01/16 15:33:56 INFO repl.SparkILoop: Created sql context (with Hive support)..SQL context available as sqlContext.scala>

 

After opening spark shell, you can write a simple program, say hello to the world

scala> println("helloworld")helloworld

Let's take a look at the spark web management interface. We can see that there is an additional Workders and Running Applications information.

So far, Spark's pseudo-distributed environment has been set up,

Note the following:

1. The Maven and SBT mentioned above are non-essential and only for subsequent source code compilation. Therefore, if you just build a Spark environment, you do not need to download Maven and SBT.

2. the pseudo-distributed environment of Spark is actually the basis of the cluster. You only need to modify a few items and copy them to the slave node. In view of the limited space, you can try again later.

For more Spark tutorials, see the following:

Install and configure Spark in CentOS 7.0

Spark1.0.0 Deployment Guide

Install Spark0.8.0 in CentOS 6.2 (64-bit)

Introduction to Spark and its installation and use in Ubuntu

Install the Spark cluster (on CentOS)

Hadoop vs Spark Performance Comparison

Spark installation and learning

Spark Parallel Computing Model

Spark details: click here
Spark: click here

This article permanently updates the link address: