Spark's RDD persistent, broadcast, accumulator

The RDD persistence, broadcast, accumulator essentially deals with how the RDD data is stored, and the RDD involves persist or checkpoint when building an efficient algorithm, as well as a broadcast and accumulator that can be tested by Spark-shell for small functions Spark-shell itself is a program launched by Spark's release package that can be used to write code directly, and Spark-shell will run the code directly.

1.1. RDD Persistent Combat

Consider persistence from 2 levels:

1) How to save results when operating the RDD ( part of Action)

Use the following Spark-shell in combat:

1.1.1.  Action-level operations to persist-start the operating environment

we use based on Hadoop 's HDFs file system, so just start Hadoop 's hdfs :

To see if the launch was successful:

Start Spark cluster:

To start the Log Manager:

start Spark-shell:

Build a RDD:

1.1.1.1.           Reduce

Perform a Action actions:

1.1.1.2.           Map

1.1.1.3.           Collect

the various The results on the Executor are collected after the cluster terminal is displayed.

we can look at the source of collect: (rdd.scala 926 line )

/**
* Return An array, contains all of the elements in this RDD.
*/
defCollect(): array[T] = withscope {
Valresults = sc.runjob ( This, (iter:Iterator[T]) = Iter.toarray)
Array. concat (Results: _*)
}

Collect Returns a araaythat works in a flowchart such as:

If you want to see the results on the command terminal, you must use collect.

generally actions at the action level will trigger sc.runjob:

1.1.1.4.           Count

1.1.1.5.           Take

1.1.1.6.           Countbykey

/**
* Count the number of elements for each key, collecting the results to a local Map.
*
* Note that this method should only being used if the resulting map is expected to be small, as
* The whole thing is loaded into the driver ' s memory.
* To handle very large results, consider using rdd.mapvalues (_ = 1L). Reducebykey (_ + _), which
* Returns an rdd[t, Long] instead of a map.
*/
defCountbykey(): map[K, Long] = self.withscope {
Self.mapvalues (_ =1L). Reducebykey (_ + _). Collect (). Tomap
}

Count The occurrences of each key:

1.1.1.7.           Saveastextfile

Saveastextfile can write data directly to HDFS .

2) in the implementation of the algorithm to the cache,persist, and another is a checkpoint

1.1.2.  throughPersistfor Persistence

by default, Spark puts its data in memory and fits in memory for high-speed iterations. You need to persist in the case of:

1) In a step calculation is particularly time consuming

2) Calculation of the length of the chain is particularly long

3)checkpoint 's RDD must also be persisted ( before checkpoint persist)

4) after shuffle

5) before shuffle (the framework defaults to help us persist the data to the local disk), if shuffle is wrong, all the parent RDD will have to be recalculated, at a great cost.

650) this.width=650; "Width=" 698 "height=" "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ( "/e/u261/themes/default/images/word.gif") no-repeat center;border:1px solid #ddd; "alt=" Spacer.gif "/>

650) this.width=650; "Width=" 697 "height=" 186 "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/images/localimage.png") no-repeat center;border:1px solid #ddd; "alt=" Spacer.gif "/>

If you find that memory is often insufficient or appears OOM, a very good way is the MEMORY of the content serialization, of course, when the use of data need to deserialize, anti-sequence words are very CPU-intensive ;

The following two are prioritized into memory, and then put to disk if the memory is not in place:

ValMemory_and_disk= NewStoragelevel (true, True, false, true )
Valmemory_and_disk_2= NewStoragelevel (true, True, false, true,2)

650) this.width=650; "Width=" 697 "height=" "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ( "/e/u261/themes/default/images/word.gif") no-repeat center;border:1px solid #ddd; "alt=" Spacer.gif "/>

650) this.width=650; "Width=" 698 "height=" "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ( "/e/u261/themes/default/images/word.gif") no-repeat center;border:1px solid #ddd; "alt=" Spacer.gif "/> There's no difference between the two execution times above, but if you first cache and count

650) this.width=650; "Width=" 698 "height=" "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ( "/e/u261/themes/default/images/word.gif") no-repeat center;border:1px solid #ddd; "alt=" Spacer.gif "/>

650) this.width=650; "Width=" height= "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ( "/e/u261/themes/default/images/word.gif") no-repeat center;border:1px solid #ddd; "alt=" Spacer.gif "/> The execution time will increase by nearly twice.

A conclusion is also drawn: Cache must not immediately have other operators!!!

650) this.width=650; "Width=" 697 "height=" "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/images/localimage.png") no-repeat center;border:1px solid #ddd; "alt=" Spacer.gif "/>

The cache is not an Actionbecause it does not perform a job. persist is a lazy level,unpersist is eager level and can be used after the cache unpersist Clears the cache. the cache is only placed in memory, and persist can be memory or disk.

1.2. Spark Broadcast combat

It is important to build the algorithm, whether it is to reduce the amount of data transmitted over the network, to improve the efficiency of memory usage, or to speed up the operation of the program, broadcasting is of paramount importance to us.

Why do I need a broadcast?

broadcasts are made by Driver the global system variables that are distributed to all Executor memory levels of the current application , and threads in the thread pool in Executor share the global variable. Greatly reduces the network transmission (otherwise each Task should be purely one time) and greatly saves memory, of course, it also increases the efficiency of the CPU effectively.

1.3. Spark Accumulator Combat

the accumulator is a global pointer step-down variable in the Spark cluster, and only the contents of the accumulator can be modified in all Executor . That is, only add the contents of the accumulator, in the Executor can not read the contents of the accumulator, in the Driver can read the contents of the accumulator.

Note:

Data from: Dt_ Big Data DreamWorks (the fund's legendary action secret course)-IMF

For more private content, please follow the public number: Dt_spark

If you are interested in big data spark, you can listen to it free of charge by Liaoliang teacher every night at 20:00 Spark Permanent free public class, address yy room Number: 68917580

Life was short,you need to Spark.

This article from "Cathy" blog, declined reprint!

Spark's RDD persistent, broadcast, accumulator