Customizing the spark streaming receiver based on xmemcached protocol Message Queuing

Source: Internet
Author: User
Tags message queue

Although spark streaming defines commonly used receiver, it is sometimes necessary to customize its own receiver. For a custom receiver, you only need to implement the receiver abstract class of spark streaming. The implementation of receiver requires simply implementing two methods:

1, OnStart (): Receive data.

2, OnStop (): Stop receiving data. General OnStart () should not be blocked and should start a new thread for complex data reception. The OnStop () method is responsible for ensuring that the threads that receive the data are stopped, Receiveris closed when called, can do some close work.     The thread that is responsible for receiving the data can use isstopped () to determine whether to stop the data reception. For the received data, it needs to be stored in the spark frame, using the store () method. The receiver abstraction class provides 4 store () methods, which can be used for storage of: 1,

Single Small Data

2. Block data in array form

3,bytebuffer form of block data

4,iterator form of block data

The implementation of these 4 store () methods is to pass data directly to Receiversupervisor for storage. So to customize a receiver, just create a data-receiving thread in OnStart () and store () to the Spark streamimg framework when you receive the data.

The following code is a spark streaming receiver based on the XMEMCACHED protocol message queue:

Import Fqueue.fqueuereceiverimport Org.apache.spark.Loggingimport Org.apache.spark.storage.StorageLevelimport Org.apache.spark.streaming.receiver.Receiverclass Fqueuestreamingreceiver (Val address:string, Val Connectionpoolsize:int, Val timeout:int) extends receiver[string] (storagelevel.memory_and_disk_2) with Logging {Priva  Te var receiver:option[fqueuereceiver] = None def onStart () {New Thread ("Socket receiver") {override def run () {Receive ()}}. Start ()} def onStop (): unit = {receiver foreach {_.stop ()}} private Def receive (): unit = {val Fqueuerecei ver = new Fqueuereceiver (address, connectionpoolsize, timeOut) receiver = Some (fqueuereceiver) receiver foreach {_. Connect ()} try {var stop = False while (!isstopped () &&!stop) {val data = fqueuereceive R.dequeue ("track_bodao2015*") data match {case Some (str) = + store (str) case None + = Thread     . Sleep (//stop) = true} } receiver foreach {_.stop ()}} catch {case e:exception = println ("Get Data from Fqueue err! Pleace sure the server is Live ") println (e.getmessage) println (e.getstacktracestring) receiver FOREAC h {_.stop ()}}}}

After you have customized receiver for spark streaming , you can use it in your app:

def main (args:array[string]) {    new Thread ("Fqueue sender") {      override def run () {senddata ()}    }.start () 
   
    val config = new sparkconf (). Setappname ("Testfqueue"). Setmaster ("local[2]")    val ssc = new StreamingContext (config , Seconds (5))    val lines = Ssc.receiverstream (new Fqueuestreamingreceiver ("localhost:18740", 4, 4000))    Lines.print ()    Ssc.start ()    ssc.awaittermination ()  }
   

Link:

Http://spark.apache.org/docs/latest/streaming-custom-receivers.html

Xmemcached-based Message Queuing

Receiver Realization of the source code in this paper

If you like, send a Star (⊙o⊙) to the GitHub project ... Your start is my motivation!

Customizing the spark streaming receiver based on xmemcached protocol Message Queuing

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.