Spark Learning six: Spark streaming

Spark Learning six: Spark streaming

tags (space delimited): Spark

• Spark learning six spark streaming
  • An overview
  • Case study of two enterprises
  • How the three spark streaming works
  • Application of four textfilestreaming
  • How to develop in four enterprises
  • Five summary

One, overview

A simple example
1, installing NC

-lk9999

2. Start the application

./bin/run-example streaming.NeworkWordCount9999

Second, enterprise case analysis

Demand:
Real-time statistics on the last two hours of website access,
PV,UV, Region

5 minutes to execute once

10:00
8:00-10:00 Window 24
10:05
8:05-10:05 Window 24

StreamingContext(sc, Mintuns(5))DStream.window(Mintuns(24 * 5), Mintuns(2 * 5))

Code implementation

<dependency>    <groupId>org.apache.spark</groupId>    <artifactId>spark-streaming_2.10</artifactId>    <version>1.3.0</version></dependency>

How to execute a Scala code in Spark-shell

> :load /opt/cdh5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/HdfsWordCount.scala

Third, how Spark streaming works

Four, the application of textfilestreaming

1, prepare the data

bin/hdfs dfs -put wordcount.txt /spark/streaming

2. Launch the Spark app

bin/spark-shell--master local[2]

3, writing code

import org. Apache. Spark. _import org. Apache. Spark. Streaming. _import org. Apache. Spark. Streaming. StreamingContext. _val SSC = new StreamingContext (SC, Seconds ( -)) Val lines = SSC. Textfilestream("Hdfs://study.com.cn:8020/myspark") Val words = lines. FlatMap(_. Split(",")) Val pairs = words. Map(Word = Word,1)) Val wordcounts = pairs. Reducebykey(_ + _) wordcounts. Print() SSC. Start() SSC. Awaittermination()

4, test results

Four, the development of enterprises in the way:

1,idea encoding, playing into jar package execution
2, put it in the script and execute it.

touch test.scala

import org. Apache. Spark. _import org. Apache. Spark. Streaming. _import org. Apache. Spark. Streaming. StreamingContext. _val SSC = new StreamingContext (SC, Seconds ( -)) Val lines = SSC. Textfilestream("Hdfs://study.com.cn:8020/myspark") Val words = lines. FlatMap(_. Split(",")) Val pairs = words. Map(Word = Word,1)) Val wordcounts = pairs. Reducebykey(_ + _) wordcounts. Print() SSC. Start() SSC. Awaittermination()

:load /opt/app/spark-1.3.0-bin-2.5.0/test/test.scala

Five, summary

Spark Learning six: Spark streaming