Spark Streaming and Flume-ng docking experiment (good text forwarding)

Forwarded from the Mad Blog

Http://www.cnblogs.com/lxf20061900/p/3866252.html

Spark Streaming is a new real-time computing tool, and it's fast growing. It converts the input stream into a dstream into an rdd, which can be handled using spark. It directly supports a variety of data sources: Kafka, Flume, Twitter, ZeroMQ, TCP sockets, etc., there are functions that can be manipulated:,,, map reduce joinwindow等。

This article will connect spark streaming and Flume-ng, then use the official built-in Javaflumeeventcount for reference, modify it and then put it on the cluster to run.

One, download spark streaming flume plug-in package, we here the Spark version is 1.0.0 (standlone), this plug-in package version selection Spark-streaming-flume_2.10-1.0.1.jar, This version fixes an important bug, refer to the following reference in the 7.

Second, the Spark's compiled jar package and the above flume plug-in, put into the project, write the following class (reference 8 in the example modified), the code is as follows:

 1 package com.spark_streaming; 2 3 Import org.apache.spark.SparkConf; 4 Import org.apache.spark.api.java.function.Function; 5 Import org.apache.spark.streaming.*; 6 Import org.apache.spark.streaming.api.java.*; 7 Import Org.apache.spark.streaming.flume.FlumeUtils; 8 Import org.apache.spark.streaming.flume.SparkFlumeEvent; 9 Public final class Javaflumeeventcount {one private javaflumeeventcount () {}13) public static void main (STR Ing[] (args) {all strings host = args[0];17 int port = integer.parseint (Args[1]); Duration Batchinterval = New Duration (Integer.parseint (args[2)); sparkconf sparkconf = new sparkconf (). Setappname ("Javaflumeeventcount"); Javastreamingcontext SSC = new Javastreamingcontext (sparkconf, batchinterval); javareceiverinputdstream<sp arkflumeevent> Flumestream = Flumeutils.createstream (SSC, host, Port), Flumestre Flumestream.count (); Am.count (). Map (new Function<long, string> () {@OverRide28 Public String Call (Long in) {"Received" + "+" + "flume events."; }31}). print (); Ssc.start (); Ssc.awaittermination (); 35}36}

The difference between this and the official is that it removes the parameter count check and adds a custom time interval (split stream), which is the third parameter. This class does not do too much processing, the introduction of the main.

Packing this class to Ifeng_spark.jar, together with Spark-streaming-flume_2.10-1.0.1.jar, is uploaded to the nodes in the Spark cluster.

Four, start flume, this flume sink to use Avro, specify a node to send to the spark cluster, we are here 10.32.21.165:11000.

V. Execute the following command under the Spark installation root directory:

./bin/spark-submit--master spark://10.32.21.165:8070--driver-memory 4G--executor-memory 4G--jars/usr/lib/spark-1. 0.0-cdh4/lib/spark-streaming-flume_2.10-1.0.1.jar,/usr/lib/flume-ng-1.4-cdh4.6.0/lib/ Flume-ng-sdk-1.4.0-cdh6.0.jar/usr/lib/spark-1.0.0-cdh4/ifeng_spark.jar--class com.spark_streaming. Javaflumeeventcount 10.32.21.165 11000 2000

The parameters in this command are explained in reference 3 below, you can also add some of the parameters, you need to be aware of the configuration of memory, self-increased memory as needed (driver, executor) to prevent oom. In addition, jars can load multiple jar packages at the same time, separated by commas. Remember to specify 3 parameters after specifying a class.

If you do not specify an SDK package for flume, the following error will be exploded:

java.lang.noclassdeffounderror:lorg/apache/flume/source/avro/avroflumeevent; no class found. This class is available in the SDK package for Flume, specifying the jar package location in the jars parameter.

There is a separate list of the jars of the business class that you define, not the jars parameter, or the error throws.

After running you can see a lot of output information, and then you can see the data of the RDD will be counted out how many lines of this rdd, such as the next, the last part of this 2 seconds (the last parameter of the above command set) statistical results:

At this point, the docking of Flume-ng with Spark was a success, just an introductory experiment. The relevant business classes can be flexibly written to realize the real-time processing of flume transmitted data.

Spark streaming and some data transmission tool docking can achieve real-time processing purposes.

Reference:

1, https://spark.apache.org/docs/0.9.0/streaming-programming-guide.html

2, http://www.cnblogs.com/cenyuhai/p/3577204.html

3, http://blog.csdn.net/book_mmicky/article/details/25714545, important parameter explanation

4, http://blog.csdn.net/lskyne/article/details/37561235, this is an example

5, http://search.maven.org/#search%7cga%7c1%7cg%3a%22org.apache.spark%22%20, spark-flume plugin download

6, Http://outofmemory.cn/spark/configuration, spark some configurable parameter description

7, https://issues.apache.org/jira/browse/SPARK-1916, this is 1.0.1 previous version of SPARK streaming and flume docking a bug information

8, Https://github.com/apache/spark/tree/master/examples/src/main/java/org/apache/spark/examples/streaming, Here are some examples of the Java version of Spark streaming, with a flume

Spark Streaming and Flume-ng docking experiment (good text forwarding)