Spark Advanced sequencing Complete decryption

　Sort of, really very important!

Rdd.scala (source code)

In its, no listing sort, not saying it doesn't matter!

1. Basic Sorting Algorithm Combat

2, two-time sorting algorithm combat

3, higher ranking algorithm

4, higher ranking algorithm

1. Basic Sorting Algorithm Combat

Start the HDFs cluster

[Email protected]:/usr/local/hadoop/hadoop-2.6.0$ sbin/start-dfs.sh

Start the Spark cluster

[Email protected]:/usr/local/spark/spark-1.5.2-bin-hadoop2.6$ sbin/start-all.sh

Start Spark-shell

[Email protected]:/usr/local/spark/spark-1.5.2-bin-hadoop2.6/bin$ ./spark-shell--master spark:// sparksinglenode:7077--executor-memory 1g

scala> sc.setloglevel ("WARN")//filter Log Reminders

scala> sc.textfile ("/readme.md"). FlatMap (_.split (")"). Map (Word = = (word,1)). Reducebykey (_+_,1). Map (pair = = (pair._2,pair._1)). Sortbykey (FALSE). map (pair = = (pair._2,pair._1)). Collect

Scala> sc.textfile ("/readme.md"). FlatMap (_.split (")"). Map (Word = (word,1)). Map (pair = > (pair._2,pair._1)). Sortbykey (FALSE). map (pair = = (pair._2,pair._1)). Collect
res2:array[(String, Int)] = Array (("", "the,21"), (spark,14), (to,14), (for,12), (a,10), (and,10), (##,8), (run,7), (i s,6), (on,6), (can,6), (of,5), (also,5), (in,5), (if,4), (or,4), (hadoop,4), (with,4), (you,4), (build,3), [including,3], (please,3), (use,3), (particular,3), (documentation,3), (example,3), (an,3), (you,3), (building,3), (that,3), guidance , 3), (for,2), (this,2), (hive,2), (to,2), (sparkpi,2), (refer,2), (interactive,2), (be,2), (./bin/run-example,2), (1000: , 2), (tests,2), (examples,2), (at,2), (using,2), (shell,2), (class,2), (' Examples ', 2), (set,2), (hadoop,,2), (cluster,2) , (supports,2), (python,2), (general,2), (locally,2), (following,2), (which,2), (should,2), ([project,2), (do,2), how,2 ), (it,2), (scala,2), (detailed,2), (return,2), (one,2), (python,,2), (SQL ...
Scala>

Then, it can be seen that Sortbykey (false) is sorted by key and descending

Sortbykey Source

/**
* Sort the RDD by key, so, each partition contains a sorted range of the elements. Calling
* ' collect ' or ' save ' on the resulting RDD would return or output an ordered list of records
* (In the "save" case, they'll be written to multiple ' part-x ' files in the filesystem, in
* Order of the keys).
*/
Todo:this currently doesn ' t work in P other than tuple2!
def sortbykey (Ascending:boolean = true, Numpartitions:int = self.partitions.length)
    : rdd[(K, V)] = Self.withscope
{
  Val part = new Rangepartitioner (numpartitions, self, ascending)
  New Shuffledrdd[k, V, v] (self, part)
    . setkeyordering (if (ascending) ordering else ordering.reverse)
}

Thus, it can be seen that, once sorted, generates SHUFFLEDRDD.

Why didn't I show it?

What sort of rangpartition is it?

OK, the basic sorting algorithm is in combat.

2, two-time sorting algorithm combat

The so-called, two-order , refers to the sort of time to consider two dimensions.

For example, in the first column, in descending order, the first column of the key is the same, then, how to arrange it? Then, consider the second column, sorted in descending order. That is, the order was used two times .

Get ready

"Data file Input"

2 3

4 1

3 2

4 3

: U

2 1

"Run result output"

2 1

2 3

3 2

4 1

4 3

: U

If you are going to a big company, you have to master 5 dimensions, even 8 dimensions, not just 2 dimensions. Come on! Zhouls.

Here, use the Scala IDE for Eclipse to write,

Initial use of Scala IDE for eclipse download, installation, and WordCount (local mode and cluster mode)

　　

Secondarysortkey.java

Package com.zhouls.spark.SparkApps.cores;

Import java.io.Serializable;
Import scala.math.Ordered;

public class Secondarysortkey implements ordered<secondarysortkey>,serializable{
private int first;
private int second;

@Override
public boolean $greater (Secondarysortkey arg0) {
TODO auto-generated Method Stub
return false;
}

@Override
public boolean $greater $eq (Secondarysortkey arg0) {
TODO auto-generated Method Stub
return false;
}

@Override
public boolean $less (Secondarysortkey arg0) {
TODO auto-generated Method Stub
return false;
}

@Override
public boolean $less $eq (Secondarysortkey arg0) {
TODO auto-generated Method Stub
return false;
}

@Override
public int Compare (Secondarysortkey arg0) {
TODO auto-generated Method Stub
return 0;
}

@Override
public int CompareTo (Secondarysortkey arg0) {
TODO auto-generated Method Stub
return 0;
}
}

Then, modify it to what we want.

The final Secondarysortkey.java are as follows:

Package com.zhouls.spark.SparkApps.cores;

Import java.io.Serializable;
Import scala.math.Ordered;

public class Secondarysortkey implements ordered<secondarysortkey>,serializable{
private int first;
private int second;

Two order of public constructors
Public Secondarysortkey (int first,int second) {
This.first=first;
This.second=second;
}

public boolean $greater (Secondarysortkey other) {
if (This.first>other.getfirst ()) {
return true;
}else if (This.first==other.getfirst () &&this.second>other.getsecond ()) {
return true;
}
return false;
}

public boolean $greater $eq (Secondarysortkey) {
if (this. $greater (other)) {
return true;
}else if (This.first==other.getfirst () &&this.second==other.getsecond ()) {
return true;
}
return false;
}

public boolean $less (Secondarysortkey other) {
if (This.first<other.getfirst ()) {
return true;
}else if (This.first==other.getfirst () &&this.second<other.getsecond ()) {
return true;
}
return false;
}

public boolean $less $eq (Secondarysortkey) {
if (this. $less (other)) {
return true;
}else if (This.first==other.getfirst () &&this.second==other.getsecond ()) {
return true;
}
return false;
}

public int Compare (Secondarysortkey other) {
if (This.first-other.getfirst ()!=0) {
return This.first-other.getfirst ();
}else{
return This.second-other.getsecond ();
}
}

public int CompareTo (Secondarysortkey other) {
if (This.first-other.getfirst ()!=0) {
return This.first-other.getfirst ();
}else{
return This.second-other.getsecond ();
}
}

public int hashcode () {
final int prime = 31;
int result = 1;
result = Prime * result + first;
result = Prime * result + second;
return result;
}

public boolean equals (Object obj) {
if (this = = obj)
return true;
if (obj = = null)
return false;
if (GetClass ()! = Obj.getclass ())
return false;
Secondarysortkey other = (secondarysortkey) obj;
if (first! = Other.first)
return false;
if (second! = Other.second)
return false;
return true;
}

public int GetFirst () {
return first;
}

public void Setfirst (int first) {
This.first = First;
}

public int Getsecond () {
return second;
}

public void Setsecond (int second) {
This.second = second;
}
}

Go on...

Reference:

Http://blog.sina.com.cn/s/blog_4a7854d90102ws97.html

Spark Advanced sequencing Complete decryption