Using idea to develop spark submit remote cluster execution

Development environment Operating system: Windows Development tools: IntelliJ idea 14.1.1 need to install Scala plugin compilation environment: JDK 1.7 Scala 2.10.4 using idea to develop SPARK application 1: New Scala project:

2: Choose the Scala version, need the Scala version of Spark corresponding to click on the Scala SDK's Create button, pop up the following dialog box, select the Red box
3: Add spark jar as library I directly copy all the jars inside the Spark installation directory under Lib as library.4: write the Spark app

  
 

   
  

  
import scala.math.random
   
  

  
import org.apache.spark._
   
  

  
/** Computes an approximation to pi */
   
  

  
object SparkPi {
   
  

  
 def main(args: Array[String]) {
   
  

  
 val conf  =    new   sparkconf   ().  setappname   (  "Spark Pi"   setmaster   (  "spark://192.168.1.88:7077"    
   
  

  
 set("spark.driver.host","192.168.1.129").setJars(List("D:\\IdeaProjects\\scalalearn\\out\\artifacts\\scalalearn\\scalalearn.jar"))
   
  

  
 val spark = new SparkContext(conf)
   
  

  
 val Slices  =    if    ( args   length  >    0  )   args  Span class= "pun" > ( 0   toint  else    2   
   
  

  
 val n = 100000 * slices
   
  

  
 val count = spark.parallelize(1 to n, slices).map { i =>
   
  

  
 val x = random * 2 - 1
   
  

  
 val y = random * 2 - 1
   
  

  
 /span>if     ( x  *  x  +   y  *  y  <    1  )    1    else    0   
   
  

  
 }.reduce(_ + _)
   
  

  
 println("Pi is roughly " + 4.0 * count / n)
   
  

  
 spark.stop()
   
  

  
 }
   
  

  
}
  
 

Submitting an app to a remote spark cluster there is a segment in the above codeSetjars(List("D:\\ideaprojects\\scalalearn\\out\\artifacts\\scalalearn\\scalalearn.jar"))here is the absolute path to the jar package that is currently being compiled by the project. Click File->project Structure, pop up the following dialog box to operate




Okay, now it's OK.


From for notes (Wiz)

Using idea to develop spark submit remote cluster execution