Eclipse 編寫的spark版WordCount運行在spark上

標籤：des   blog   http   ar   sp   on   div   2014   art   

1、代碼編寫

 if (args.length != 3 ){
      println("usage is org.test.WordCount <master> <input> <output>")
      return
    }
    val sc = new SparkContext(args(0), "WordCount",
    System.getenv("SPARK_HOME"), Seq(System.getenv("SPARK_TEST_JAR")))
    val textFile = sc.textFile(args(1))
    val result = textFile.flatMap(line => line.split("\\s+"))
        .map(word => (word, 1)).reduceByKey(_ + _)
    result.saveAsTextFile(args(2))

2、匯出jar包，這裡我命名為WordCount.jar

3、運行

bin/spark-submit --master spark://master:7077 /home/zkpk/Desktop/WordCount.jar hdfs://master:9000/dedup_in hdfs://master:9000/dedup_out

先記錄下，第三部還沒執行，抽空試試

 

 

Eclipse 編寫的spark版WordCount運行在spark上