Spark Learning III: Installing and Importing source code for spark schedule and idea
tags (space delimited): Spark
- Spark learns to install and import source code for three spark schedule and idea
- Data location during an RDD operation
- Two spark Schedule
- Three idea import spark source
One, the data location during the RDD operation
[hadoop001@xingyunfei001 spark-1.3.0-bin-2.5.0]$ bin/spark-shell --master local[2]
Val Rdd = sc.textfile ("xx", 2)
Rdd.flatmap (. Split ("\ T")). Map ((, 1)). Reducebykey ((_ + _)). Saveastextfile
Stage-01
Part-01-Task-01:rdd.flatmap (. Split ("\ T")). Map ((, 1)). reduce
Part-02-Task-02:rdd.flatmap (. Split ("\ T")). Map ((, 1)). reduce
Stage-02
Part-01-Task-01:bykey ((_ + _)). Saveastextfile
Part-02-Task-02:bykey ((_ + _)). Saveastextfile
Rdd.cache
Two, Spark Schedule
Third, Idea import spark source code
1,bin/idea.sh Open Idea (progress bar click Cancel when entering)
2, integrated maven
3, import the source code
Spark Learning III: Installing and Importing source code for spark schedule and idea