SparkStreaming programming guide

SparkStreaming is the core api of Spark. It supports high throughput and Fault-Tolerant Real-time stream data processing. It can accept data sources from Kafka, Flume, Twitter, ZeroMQ, and TCPSocket, and use simple api functions such as map, reduce, join, and window, you can also directly use built-in machine learning algorithms and graph algorithms.

Spark Streaming is a core api of Spark. It supports high throughput and Fault-Tolerant Real-time stream data processing. It can accept data sources from Kafka, Flume, Twitter, ZeroMQ, and TCP Socket, and use simple api functions such as map, reduce, join, and window, you can also directly use built-in machine learning algorithms and graph algorithms.

Spark Streaming is a core api of Spark. It supports high throughput and Fault-Tolerant Real-time stream data processing.

It can accept data sources from Kafka, Flume, Twitter, ZeroMQ, and TCP Socket, and use simple api functions such as map, reduce, join, and window, you can also directly use built-in machine learning algorithms and Graph Algorithm packages to process data.

Its workflow is as shown in the following figure. After receiving real-time data, it divides the data into batches, and then passes it to Spark Engine for processing to generate the results of this batch.

It supports data streams called Dstream and directly supports data sources of Kafka and Flume. Dstream is a continuous RDDs. The following is an example to help you understand Dstream.

A Quick Example

// Create StreamingContext, one batch per second val ssc = new StreamingContext (sparkConf, Seconds (1); // configure a DStream to connect to the listening port: Address val lines = ssc. socketTextStream (serverIP, serverPort); // execute the Split operation val words = lines for each row of data. flatMap (_. split (""); // count the number of words. val pairs = words. map (word => (word, 1); val wordCounts = pairs. reduceByKey (_ + _); // output result wordCount. print (); ssc. start (); // start ssc. awaitTermination (); // exit after calculation.

The specific code can access this page:

Https://github.com/apache/incubator-spark/blob/master/examples/src/main/scala/org/apache/spark/streaming/examples/NetworkWordCount.scala

If you have installed a friend of Spark, we can use the following example to try it.

First, start Netcat. This tool exists in Unix-like systems and is a simple data server.

Run the following command to start Netcat:

$ Nc-lk 9999
Start example

$./Bin/run-example org. apache. spark. streaming. examples. NetworkWordCount local [2] localhost 9999
Enter hello world on the Netcat side. Check the Spark side.

# TERMINAL 1:# Running Netcat$ nc -lk 9999hello world...# TERMINAL 2: RUNNING NetworkWordCount or JavaNetworkWordCount$ ./bin/run-example org.apache.spark.streaming.examples.NetworkWordCount local[2] localhost 9999...-------------------------------------------Time: 1357008430000 ms-------------------------------------------(hello,1)(world,1)...

Basics
The following figure shows how to write code. Wow!

First, add the following information to the SBT or Maven project:

GroupId = org. apache. spark
ArtifactId = spark-streaming_2.10
Version = 0.9.0-incubating

// You need to use the data Source, but also add the corresponding dependency Source ArtifactKafka spark-streaming-kafka_2.10Flume spark-streaming-flume_2.10Twitter spark-streaming-twitter_2.10ZeroMQ spark-streaming-zeromq_2.10MQTT spark-streaming-mqtt_2.10

The next step is instantiation.

New StreamingContext (master, appName, batchDuration, [sparkHome], [jars])
This is the operation of DStream in the previous example.

Input Sources
In addition to sockets, we can also create Dstream

StreamingContext. fileStream (dataDirectory)

There are three key points:

(1) The file formats in dataDirectory are the same.

(2) All files created in this directory are created by moving or renaming them.

(3) Once the file is in, it cannot be changed.

Suppose we want to create a Dstream of Kafka.

Import org. apache. spark. streaming. kafka ._
KafkaUtils. createStream (streamingContext, kafkaParams ,...)

If we need a handler for a custom stream, you can view the https://spark.incubator.apache.org/docs/latest/streaming-custom-receivers.html

Operations
For Dstream, we can perform two operations: transformations and output.

Transformations

Transformation Meaningmap (func) executes the func method flatMap (func) similar to the map function for each element, but can map to 0 + output filters (func) to filter repartition (numPartitions) and add partitions, increase the degree of parallelism (otherStream) Merge the number of count () statistical elements of two streams reduce (func) aggregate the elements in RDDs, two input parameters, one output parameter countByValue () is used for type statistics. When the type of a Dstream element is K, calling it will return a new Dstream containing key-value pairs, long is the frequency at which each K appears. Performancebykey (func, [numTasks]) executes the func function for each key of A (K, V) type Dstream. The default value is local, which is two threads, cluster is eight threads. You can also specify numTasks join (otherStream, [numTasks]) to connect (K, V) and (K, W) Dstream into one (K, (V, W) The new Dstream cogroup (otherStream, [numTasks]) connects (K, V) and (K, W) Dstream into a (K, seq [V], Seq [W]) New Dstream transform (func) conversion operation, convert the original RDD to a new RDDupdateStateByKey (func) through func) use func for key to update the status and value. You can set the state value to any value.

UpdateStateByKey Operation
With this operation, we want to save its status information and update it continuously. There are two steps to use it:

(1) define the status, which can be any data type

(2) define the status update function to change the previous status

The following is an example:

def updateFunction(newValues: Seq[Int], runningCount: Option[Int]): Option[Int] = {    val newCount = ...  // add the new values with the previous running count to get the new count    Some(newCount)}

It can be used in Dstream containing (word, 1), for example, example

Val runningCounts = pairs. updateStateByKey [Int] (updateFunction _)

It calls the UPDATE function for each word. newValues is the latest value and runningCount is the previous value.

Transform Operation
Like transformWith, you can perform RDD-> RDD operations on a Dstream. For example, we want to perform join operations on the RDD and another dataset in the Dstream, however, Dstream APIs are not directly exposed. We can use the transform method to perform this operation. The following is an example:

Val spamInfoRDD = sparkContext. hadoopFile (...) // RDD containing spam information

Val cleanedDStream = inputDStream. transform (rdd => {
Rdd. join (spamInfoRDD). filter (...) // Join data stream with spam information to do data cleaning
...
})

In addition, we can also use machine learning algorithms and graph algorithms.

Window Operations
,

For example, in the previous word count example, we want to calculate the total number of words in the last 30 seconds every 10 seconds.

You can use the following statement:

// Reduce last 30 seconds of data, every 10 seconds
Val calculated wedwordcounts = pairs. performancebykeyandwindow (_ + _, Seconds (30), Seconds (10 ))

Two windows parameters are mentioned here:

(1) window length: The window length is 30 seconds, and the data of the last 30 seconds

(2) slice interval: calculation interval

In this example, we can calculate the sliding data periodically.

Below are some operation functions of the window, but I still cannot understand the concept of the window. Meaning will not translate it and will delete it directly.

Transformation                                                                              Meaningwindow(windowLength, slideInterval)     countByWindow(windowLength, slideInterval)     reduceByWindow(func, windowLength, slideInterval)     reduceByKeyAndWindow(func, windowLength, slideInterval, [numTasks])     reduceByKeyAndWindow(func, invFunc, windowLength, slideInterval, [numTasks])    countByValueAndWindow(windowLength, slideInterval, [numTasks])  

Output Operations
Output Operation Meaning
Print () to the console
ForeachRDD (func) executes func for each RDD in Dstream and saves it to the external system.
SaveAsObjectFiles (prefix, [suffix]) saves the stream content as SequenceFile, file name: "prefix-TIME_IN_MS [. suffix]".
SaveAsTextFiles (prefix, [suffix]) saves the content of the stream as a text file named "prefix-TIME_IN_MS [. suffix]".
SaveAsHadoopFiles (prefix, [suffix]) saves the stream content as a hadoop file named "prefix-TIME_IN_MS [. suffix]".

Persistence
RDD in Dstream can also call the persist () method to save it in memory. However, for window and state-based operations, performancebywindow, performancebykeyandwindow, and updateStateByKey are implicitly saved, the system has automatically saved it.

Data received from the network (such as, Kafka, Flume, sockets, etc.) is stored in two nodes by default for fault tolerance and stored in memory in a serialized manner.

RDD Checkpointing
Status operations are performed based on multiple batches of data. It includes window-based operations and updateStateByKey. Because state operations depend on the data of the previous batch, they need to accumulate metadata based on time. To clear data, it supports periodic checkpoints by saving intermediate results to hdfs. Because the check operation will cause the overhead to be saved to hdfs, set this interval with caution. For small batches of data, such as one second, the check operation will greatly reduce the throughput. However, if the check interval is too long, the task will become larger. Generally, the check interval of 5-10 seconds is suitable.

Ssc. checkpoint (hdfsPath) // you can specify the position where the checkpoint is saved.
Dstream. checkpoint (checkpointInterval) // you can specify the checkpoint interval.

For dstreams that must be set for checkpoints, such as dstreams created through updateStateByKey and performancebykeyandwindow, the default setting is at least 10 seconds.

Performance Tuning
For optimization, you can consider the following two aspects:

(1) reduce the time required to process each batch of data using cluster resources

(2) set a proper size for each batch of data

Level of Parallelism
For some distributed operations, such as performancebykey and performancebykeyandwindow, the default eight concurrent threads can increase their values through corresponding functions, or by modifying the spark parameter. default. parallelism to increase this default value.

Task Launching Overheads
Too many tasks are not good, for example, 50 tasks per second. The load for sending tasks becomes very important, and it is difficult to implement latency in seconds, of course, the batch size can be reduced through compression.

Setting the Right Batch Size
To enable the stream program to run stably on the cluster, the data processing speed should keep up with the data Inbound speed. The best way to calculate the batch size is to set the batch size to 5-10 seconds and a low data input speed. When the system is able to keep up with the data speed, we can set the size based on experience and check the log to see how long the Total delay takes. If the delay value is smaller than batch, the system can be stable. If the delay value increases continuously, the processing speed of the system cannot keep up with the data input speed.

24/7 Operation
Spark does not forget metadata by default, such as the generated RDD and stages. However, Spark Streaming is a 24/7 program, which needs to periodically clean up metadata and be set through spark. cleaner. ttl. For example, if I set it to 600, Spark will understand all the metadata and persist RDDs when it is over 10 minutes. However, this attribute must be set before SparkContext is created.

However, this value is bound to any window operation. Spark requires that the input data be persisted to the memory after expiration, so the delay value must be set to be at least the same as the maximum window operation. If the setting is small, an error is returned.

Monitoring
In addition to Spark's built-in monitoring capabilities, you can also use StreamingListener to obtain the batch processing time, query latency, and all end-to-end tests.

Memory Tuning
The default serialization method of Spark Stream is StorageLevel. MEMORY_ONLY_SER, rather than StorageLevel. MEMORY_ONLY of RDD.

By default, all persistent RDD will be excluded from the memory by Spark's LRU algorithm. If spark is set. cleaner. ttl is cleared periodically, but this parameter must be set with caution. A better way is to set spark. streaming. unpersist to true, which allows Spark to calculate which RDD needs to be persisted, which is conducive to improving GC performance.

We recommend that you use concurrent mark-and-sweep GC. Although this reduces the throughput of the system, it helps to process batches more stably.

Fault-tolerance Properties
Failure of a Worker Node
There are two failure methods:

1. Use Files on hdfs. Because hdfs is a reliable file system, no data will expire.

2. if the data source is a network, such as Kafka and Flume, data is saved to two nodes by default to prevent invalidation, but one possibility is that the node that accepts the data is suspended, data may be lost because it has not been able to copy the data to another node.

Failure of the Driver Node
To support 24/7 uninterrupted processing, Spark supports re-computing after the driver node fails. Spark Streaming periodically writes data to the hdfs System, which is the directory of the previous checkpoint. After the driver node fails, StreamingContext can be restored.

To enable a Spark Streaming program to be replied, perform the following operations:

(1) At the first startup, create StreamingContext, create all streams, and then call the start () method.

(2) If you restart after recovery, you must re-create StreamingContext through the data at the checkpoint.

The following is an actual example:

You can use StreamingContext. getOrCreate to construct StreamingContext.

// Function to create and setup a new StreamingContextdef functionToCreateContext(): StreamingContext = {    val ssc = new StreamingContext(...)   // new context    val lines = ssc.socketTextStream(...) // create DStreams    ...    ssc.checkpoint(checkpointDirectory)   // set checkpoint directory    ssc}// Get StreaminContext from checkpoint data or create a new oneval context = StreamingContext.getOrCreate(checkpointDirectory, functionToCreateContext _)// Do additional setup on context that needs to be done,// irrespective of whether it is being started or restartedcontext. ...// Start the contextcontext.start()context.awaitTermination()

In the deployment mode of stand-alone, if the driver node fails, it can be automatically restored to allow other driver nodes to replace it. This can be tested locally. The supervise mode is used when the program is submitted. After the program is submitted, use jps to view the process. If it is similar to DriverWrapper, it will be killed, if the YARN mode is used, you have to restart it in other ways.

Here, by the way, I would like to submit a program to the client. I have dropped this piece in my summary.

. /Bin/spark-class org. apache. spark. deploy. client launch [client-options] \ [application-options] cluster-url: master Address. application-jar-url: Address of the jar package, preferably on hdfs, with hdfs ://... otherwise, the directory of all nodes must have the main-class of the jar: class of the main function of the program to be released. client Options: -- memory (memory of the driver, in MB) -- cores (How many cores are allocated to your driver) -- supervise (whether to restart the application when the node fails) -- verbose (print incremental log output)

In future versions, the recoverability of all data sources will be supported.

To better understand the Failure Recovery of HDFS-based drive nodes, the following is a simple example:

Time     Number of lines in input file     Output without driver failure     Output with driver failure1    　　10    　　　　　　　　　　　　　　　　　10    　　　　　　　　　　　　　　　　102    　　20    　　　　　　　　　　　　　　　　　20    　　　　　　　　　　　　　　　　203    　　30    　　　　　　　　　　　　　　　　　30    　　　　　　　　　　　　　　　　304    　　40    　　　　　　　　　　　　　　　　　40    　　　　　　　　　　　　　　　　[DRIVER FAILS] no output5    　　50    　　　　　　　　　　　　　　　　　50    　　　　　　　　　　　　　　　　no output6    　　60    　　　　　　　　　　　　　　　　　60    　　　　　　　　　　　　　　　　no output7    　　70    　　　　　　　　　　　　　　　　　70    　　　　　　　　　　　　　　　　[DRIVER RECOVERS] 40, 50, 60, 708    　　80    　　　　　　　　　　　　　　　　　80    　　　　　　　　　　　　　　　　809    　　90    　　　　　　　　　　　　　　　　　90    　　　　　　　　　　　　　　　　9010    　100    　　　　　　　　　　　　　　　　 100    　　　　　　　　　　　　　　　100

There was an error at the time of 4. There was no output at the time of 40, 50, and 60, and it was restored at the time of 70. After the restoration, all the outputs that were not output before were output at once.

Original article address: Spark Streaming programming guide. Thank you for sharing it with me.