Spark Learning III: Installing and Importing source code for spark schedule and idea

Spark Learning III: Installing and Importing source code for spark schedule and idea

tags (space delimited): Spark

• Spark learns to install and import source code for three spark schedule and idea
  • Data location during an RDD operation
  • Two spark Schedule
  • Three idea import spark source

One, the data location during the RDD operation

[hadoop001@xingyunfei001 spark-1.3.0-bin-2.5.0]$ bin/spark-shell --master local[2]

Val Rdd = sc.textfile ("xx", 2)
Rdd.flatmap (. Split ("\ T")). Map ((, 1)). Reducebykey ((_ + _)). Saveastextfile

Stage-01
Part-01-Task-01:rdd.flatmap (. Split ("\ T")). Map ((, 1)). reduce
Part-02-Task-02:rdd.flatmap (. Split ("\ T")). Map ((, 1)). reduce
Stage-02
Part-01-Task-01:bykey ((_ + _)). Saveastextfile
Part-02-Task-02:bykey ((_ + _)). Saveastextfile

Rdd.cache

Two, Spark Schedule

Third, Idea import spark source code

1,bin/idea.sh Open Idea (progress bar click Cancel when entering)

2, integrated maven

3, import the source code

Spark Learning III: Installing and Importing source code for spark schedule and idea