Spark runs a simple demo program

Spark runs a simple demo programUsing Spark, you can start Spark-shell directly on the command line and then use Scala for data processing in Spark-shell. Now we're going to show you how to write handlers using the IDE. Prerequisite: 1, already installed spark can be run up. 2. Understand the basic Scala syntax
Okay, here's the cargo. 1, first use MAVEN to create a simple QuickStart Program 2, create a package in the app directory, create a Scala file in the package (note that the IDE needs to install the Scala plugin). Effect as shown:
You may find that there is no Scala option at the time of creation, this time you can create a new file directly, with the suffix named Scala.
3. Write a simple Scala program

Package com.xueyoucto.xueyou

Import org.apache.spark.{ sparkconf, Sparkcontext}

/**
  * Created by Administrator on 2017-01-04.
  */
Object First {
  def main (args:array[string]): Unit = {
    println (123)
    val conf = new sparkconf (). Setmast ER ("spark://192.168.1.221:7077"). Setappname ("Firstapp")
    val sc = new Sparkcontext (conf)
    val A = Sc.parallelize (List (1, 2, 3, 4))
    a.persist ();
    println (A.count ())
    println ("============================")
    a.collect (). foreach (println)
  }
}

Note that you need to introduce a jar package (there are a lot of pits here: 1, it's best not to use MAVEN's jar package, because MAVEN handles dependencies automatically, even if your version of Spark is right, your Scala version might not be right.) 2, may appear link not on spark://192.168.1.221:7077 this is not your spark boot problem, or because your spark jar package and Scala's jar package and the real environment is not the same. 3. There may be no new Sparkcontext (CONF) in the main method, this is also because of your jar package version. Here to thank Http://stackoverflow.com/questions/34518669/unable-to-connect-to-spark-master, there is a good answer in this, I cut a picture)

4, in order to solve the above problem, directly from spark to copy the jar package can be. And then introduced in the IDE, I'm using idea, introduced here:

5, now can be happy to run, this is the result of the operation.

The middle of the log is a little