The spark version of Eclipse written by WordCount runs on Spark

1. Code Writing

if (args.length! = 3) {
println ("Usage is org.test.WordCount <master> <input> <output>")
Return
}
Val sc = new Sparkcontext (args (0), "WordCount",
System.getenv ("Spark_home"), Seq (System.getenv ("Spark_test_jar")))
Val textfile = Sc.textfile (args (1))
Val result = Textfile.flatmap (line = Line.split ("\\s+"))
. Map (Word (Word, 1)). Reducebykey (_ + _)
Result.saveastextfile (args (2))

2. Export jar package, here I named Wordcount.jar

3. Operation

Bin/spark-submit--master Spark://master:7077/home/zkpk/desktop/wordcount.jar hdfs://master:9000/dedup_in hdfs:// Master:9000/dedup_out

Just record it, the third part hasn't been executed, try it out.

The spark version of Eclipse written by WordCount runs on Spark