Spark Development and operation

Environment: CentOS 6.3, Hadoop 1.1.2, JDK 1.6, Spark 1.0.0, Scala 2.10.3, Eclipse (Hadoop can not be installed)

First of all understand that Spark is the Scala project. In other words, Spark was developed in Scala.

1. Please confirm that you have properly installed spark and run, if not, please install the following link to install.

http://blog.csdn.net/zlcd1988/article/details/21177187

2. New Scala Project

Create a Scala project based on your blog below

http://blog.csdn.net/zlcd1988/article/details/31366589

3. Import Spark jar package, you can use the Spark API (this jar package is the second step in the compilation of the generated jar package)

In the imported project, right-click to select Build Path->config build Path

4. Configure SBT configuration files (similar to maven pom.xml files)

Under the project root, create the BUILD.SBT file and add the following

Name: = "Simple Project"

version: = "1.0"

scalaversion: = "2.10.3"

librarydependencies + = "Org.apache.spark "% percent" spark-core "%" 1.0.0 "

resolvers + +" Akka Repository "at" http://repo.akka.io/releases/"

5. Write a spark application

Package Spark

Import org.apache.spark.SparkConf
import org.apache.spark.SparkContext

object Simpleapp {

  def main (args:array[string]) = {

    val logFile = "/usr/lib/spark/readme.md"
    val conf = new sparkconf (). Setapp Name ("Simple Application")
    val sc = new Sparkcontext (conf)
    val logdata = Sc.textfile (LogFile, 2). Cache
    Val numas = logdata.filter (line => line.contains ("a")). Count
    val numbs = logdata.filter (line => line.contains ("B")). Count
    println ("Lines with a:%s, Lines with B:%s". Format (Numas, numbs))
  }

}

6. Packaged Deployment

Go to the project root directory

$ cd Scala
$ SBT Package
[INFO] Loading Global Plugins From/home/hadoop/.sbt/0.13/plugins
[INFO] Set current project to Simple project (in Build file:/home/hadoop/workspace/scala/)
[INFO] Compiling 1 Scala Source to/home/hadoop/workspace/scala/target/scala-2.10/classes ...
[INFO] Packaging/home/hadoop/workspace/scala/target/scala-2.10/simple-project_2.10-1.0.jar ...
[INFO] Done packaging.
[Success] Total Time:5 S, completed June, 2014 10:32:48 PM

7. Run

$ $SPARK _home/bin/spark-submit--class "SPARK. Simpleapp "--master Local[4] Target/scala-2.10/simple-project_2.10-1.0.jar

Run Result:

Lines with a:73, Lines with b:35
14/06/15 22:34:05 INFO tasksetmanager:finished TID 3 in-ms on localhost (PROGRESS:2/2)
14/06/15 22:34:05 INFO taskschedulerimpl:removed TaskSet 1.0, whose tasks have all completed, from pool