Spark runs a simple demo program

Source: Internet
Author: User
Spark runs a simple demo programUsing Spark, you can start Spark-shell directly on the command line and then use Scala for data processing in Spark-shell. Now we're going to show you how to write handlers using the IDE. Prerequisite: 1, already installed spark can be run up. 2. Understand the basic Scala syntax
Okay, here's the cargo. 1, first use MAVEN to create a simple QuickStart Program 2, create a package in the app directory, create a Scala file in the package (note that the IDE needs to install the Scala plugin). Effect as shown:
You may find that there is no Scala option at the time of creation, this time you can create a new file directly, with the suffix named Scala.
3. Write a simple Scala program
Package com.xueyoucto.xueyou

Import org.apache.spark.{ sparkconf, Sparkcontext}

/**
  * Created by Administrator on 2017-01-04.
  */
Object First {
  def main (args:array[string]): Unit = {
    println (123)
    val conf = new sparkconf (). Setmast ER ("spark://192.168.1.221:7077"). Setappname ("Firstapp")
    val sc = new Sparkcontext (conf)
    val A = Sc.parallelize (List (1, 2, 3, 4))
    a.persist ();
    println (A.count ())
    println ("============================")
    a.collect (). foreach (println)
  }
}

Note that you need to introduce a jar package (there are a lot of pits here: 1, it's best not to use MAVEN's jar package, because MAVEN handles dependencies automatically, even if your version of Spark is right, your Scala version might not be right.) 2, may appear link not on spark://192.168.1.221:7077 this is not your spark boot problem, or because your spark jar package and Scala's jar package and the real environment is not the same. 3. There may be no new Sparkcontext (CONF) in the main method, this is also because of your jar package version. Here to thank Http://stackoverflow.com/questions/34518669/unable-to-connect-to-spark-master, there is a good answer in this, I cut a picture)

4, in order to solve the above problem, directly from spark to copy the jar package can be. And then introduced in the IDE, I'm using idea, introduced here:

5, now can be happy to run, this is the result of the operation.

The middle of the log is a little

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.