"Big Data Processing Architecture" 2. Use the SBT build tool to spark cluster

we use SBT to create, test, run, and submit jobs. This tutorial will explain all the SBT commands you will use in our course. the Tools Installation page explains how to install SBT. We typically make the code and libraries into jar packages that are submitted to the spark cluster via Spark-submit.

1) Download and install:

http://www.scala-sbt.org/

2) Create the project:

For example, now the project is called "Sparksample". So

2345	CDSparksamplemkdirProjectmkdirsrc/Main/Scala

The general engineering document structure is as follows:

• project– Project Definition File
  • project/build/.scala– major Engineering definition files
  • project/build.properties– Engineering, SBT and Scala version definitions
• src/main– your app code here, different subdirectory names represent different programming languages (for example, Src/main/scala,src/main/java)
• src/main/resources– static files that you want to add to the jar package (for example, a log configuration file)
• lib_managed– the jar file that your project relies on. Added to this directory when SBT is updated
• target– the directory where the final generated files are stored (for example, generated thrift code, class file, jar file)

3) Write BUILD.SBT

Name: = "Spark Sample"
Version: = "1.0"
Scalaversion: = "2.10.3"
Librarydependencies + = "Org.apache.spark" percent "Spark-core"% "1.1.1"

It is important to note that the version used, the version of Scala and spark streaming, and so on.

http://mvnrepository.com/artifact/org.apache.spark/spark-streaming_2.10/1.4.1

4) Build the jar package.

Under Project's file directory (e.g. "sparksample")

> SBT Package

5) Submit to spark:

e.g:cd/opt/spark******

Bin/spark-submit--class "Org.apache.spark.examples.streaming.KafkaWordCount"--packages Org.apache.spark: spark-streaming-kafka_2.10:1.4.1--master local[2]  /home/ubuntu/kafkawordcount/target/scala-2.10/ Kafkawordcount_2.10-1.0.jar 10.81.52.88:9092 Tintin

Specific how to write parameters, please see the official:

Http://spark.apache.org/docs/latest/submitting-applications.html#submitting-applications

Note: The slight pit is that the calling package needs to be manually added to the---packages * * * * *.

Reference: HTTP://WWW.TUICOOL.COM/ARTICLES/AJNIVQ

Http://www.scala-sbt.org/release/docs/index.html

http://www.supergloo.com/fieldnotes/apache-spark-cluster-part-2-deploy-a-scala-program-to-spark-cluster/

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

"Big Data Processing Architecture" 2. Use the SBT build tool to spark cluster