Spark Quick Start (1)

1 Installing Spark

First, go to https://spark.apache.org/downloads.html to select the latest spark version and Hadoop version (in fact we are not using Hadoop for the time being, so any version is ok), then download the package.

After that, unzip the folder to a specific location, for example, I'll name the extracted folder and spark put it in my home folder /home/qyx so I can execute

/home/qyx/spark/bin/spark-shell

To run Spark's terminal. To avoid having to enter a long string of paths each time you open spark, you can add the Spark bin directory to the system path, for example, I wrote ~/.bashrc a file

Export path= $PATH:/home/qyx/spark/bin

So the installation process is so happy to end.

2 Installing SBT

3 Simple test examples

/* Simpleapp.scala */import org.apache.spark.SparkContextimport Org.apache.spark.sparkcontext._import Org.apache.spark.SparkConfobject Simpleapp {  def main (args:array[string]) {    val logFile = "your_spark_home/ Readme.md "//should be some file on your system    val conf = new sparkconf (). Setappname (" Simple Application ")    Val S c = new Sparkcontext (conf)    val logdata = Sc.textfile (LogFile, 2). Cache ()    val numas = logdata.filter (line = Lin E.contains ("a")). Count ()    val numbs = logdata.filter (line = Line.contains ("B")). Count ()    println ("Lines With a:%s, Lines with B:%s ". Format (Numas, numbs)  }}

4 SBT Package file

Name: = "Simple Project" version: = "1.0" scalaversion: = "2.11.5" librarydependencies + = "Org.apache.spark" percent "Spark-core" % "1.4.0"

5 to keep SBT working properly, we need to put Simpleapp.scala and SIMPLE.SBT in a typical SBT project layout folder. As a result, we can package the application code into a jar file and then run the program using the Spark-submit script.

# Your Directory layout should look like this$ find .../simple.sbt./src./src/main./src/main/scala./src/main/scala/ simpleapp.scala# Package a jar containing your application$ SBT ... [INFO] Packaging {..} /{..} /target/scala-2.10/simple-project_2.10-1.0.jar# use Spark-submit to run your application$ your_spark_home/bin/ Spark-submit   --class "Simpleapp"   --master local[4]   Target/scala-2.10/simple-project_2.10-1.0.jar ... Lines with a:46, Lines with b:23

　　

Spark Quick Start (1)