Spark Streaming source interpretation of the data to clear the inside of the complete decryption

Contents of this issue:

• Spark Streaming data cleansing principles and phenomena
• Spark Streaming data Cleanup code parsing

The Spark streaming is always running, and the RDD is constantly generated during the calculation, such as producing a bachduration per second and also producing an RDD,

In this process, in addition to the basic RDD, there are accumulators, broadcast variables, and the corresponding spark streaming also has its own object, source data and data cleansing mechanism,

Each bachduration in the run will trigger the job, and will automatically be recycled as soon as it is finished running, such as objects, data, and source data.

　　

First, the data source:

　　

Create Kafka, the operation of the source data.

　　　　

　　　　

　　　　

　　　　

Second, the output of processing data:

From the study of its life cycle, the next step, the output to consider, Foreachrdd belongs to materialized (materialization), materialization is stored on the external device.

　　　　

　　　　

　　　　

Based on the data source Kafka,dstreams over time will continue to maintain a hashmap in its own memory data structure Generatedrdd, this hashmap time window of the RDD instance,

Follow bachduration to store this rdd and remove the RDD.

　　　　

Memory cache structure, sometimes call the operation of the cache, in fact, is to mark the Dstreams, the specified Storagelevel eventually acting on the RDD, there will be related cache operation.

　　　　

Third, the process of jobgenerator cleaning:

Recurring Event generation (Message Circulator):

　　　　

　　　　

　　　　

　　　　

Event firings are in time:

　　　　

　　　　

　　　　

Spark streaming is cleaned after each job processing is completed, that is, after each bachduration processing is complete, first of all the output dstreams to clean up, and then to clean up his dependencies,

When cleaning, the default is to clean up the data, such as the RDD, and metadata metadata cleanup.

remark:

• • Data from: Liaoliang (Spark release version customization)
  • Sina Weibo:http://www.weibo.com/ilovepains

Spark Streaming source interpretation of the data to clear the inside of the complete decryption