10th lesson: Spark Streaming Source interpretation data constantly receiving full life cycle thorough research and thinking

In the previous lesson, we explained the process of receiver initiation. Receiver is initiated by the Start method of the Receiversupervisor:

/** Start the Supervisor */def start () {OnStart () Startreceiver ()}

The Receiversupervisor OnStart () method is called first,

Override protected Def onStart () {registeredblockgenerators.foreach {_.start ()}}

The registeredblockgenerators is assigned when the Receiversupervisor is instantiated:

Private Val defaultblockgenerator = Createblockgenerator (defaultblockgeneratorlistener) override Def Createblockgenerator (blockgeneratorlistener:blockgeneratorlistener): Blockgenerator = {//Cleanup BlockGenerators t Hat has already been stopped registeredblockgenerators--= registeredblockgenerators.filter{_.isstopped ()} Val NewBlo Ckgenerator = new Blockgenerator (Blockgeneratorlistener, Streamid, env.conf) registeredblockgenerators + = Newblockgenerator Newblockgenerator}

Call the Start method of Blockgenerator

/** Start block generating and pushing threads. */def start (): Unit = synchronized {if (state = = Initialized) {state = Active Blockintervaltimer.start () block Pushingthread.start () loginfo ("Started blockgenerator")} else {throw new Sparkexception (S "Cannot start Bloc Kgenerator as its not in the Initialized state [state = $state] ")}}

Blockintervaltimer is a timer that calls the Updatecurrentbuffer function when it's time.

Private Val Blockintervaltimer = new Recurringtimer (clock, Blockintervalms, Updatecurrentbuffer, "Blockgenerator")

Time interval default 200 milliseconds

Private Val Blockintervalms = conf.gettimeasms ("Spark.streaming.blockInterval", "200ms")

Blockpushingthread is a thread that continually writes data to Blockmanager

Private val blockpushingthread = new thread ()  { override def run ( )  { keeppushingblocks ()  } }/** Keep pushing blocks to the  Blockmanager. */private def keeppushingblocks ()  {  loginfo ("Started block  pushing thread ")   def areblocksbeinggenerated: boolean = synchronized  {    state != stoppedgeneratingblocks  }  try {     // while blocks are being generated, keep polling  for to-be-pushed blocks and push them.    while  ( areblocksbeinggenerated)  {      option (Blocksforpushing.poll (10,  timeunit.milliseconds))  match {        case some (block)  => pushblock (bLock)         case None =>       }    }    // at this point, state is  stoppedgeneratingblock. so drain the queue of to-be-pushed blocks.     loginfo ("pushing out the last "  + blocksforpushing.size ( )  +  " blocks")     while  (!blocksforpushing.isempty)  {       val block = blocksforpushing.take ()        Logdebug (S "pushing block  $block")       pushblock (block)        loginfo ("blocks left to push "  + blocksforpushing.size () )     }    loginfo ("Stopped block pushing thread")    } catch&nbSp {    case ie: interruptedexception =>       loginfo ("block pushing thread was interrupted")     case e:  exception =>      reporterror ("error in block  Pushing thread ",  e)   }}

As can be seen from the code, each 10ms removes all blocks from the blocksforpushing queue, calling the Pushblock method

Private def pushblock (Block:block) {Listener.onpushblock (block.id, Block.buffer) Loginfo ("pushed block" + block.id)}

The listener here are in Receiversupervisorimpl.

Private Val Defaultblockgeneratorlistener = new Blockgeneratorlistener {def onadddata (Data:any, metadata:any): Unit = {} def ongenerateblock (blockid:streamblockid): Unit = {} def onError (message:string, throwable:throwable) {repo Rterror (Message, Throwable)} def onpushblock (Blockid:streamblockid, arraybuffer:arraybuffer[_]) {Pusharraybuffer ( ArrayBuffer, None, Some (Blockid))}}

So the Pusharraybuffer method is called, and the following method is called:

/** store block and report it to driver */def pushandreportblock (     receivedblock: receivedblock,    metadataoption: option[ any],    blockidoption: option[streamblockid]  )  {  val  blockid = blockidoption.getorelse (Nextblockid)   val time =  System.currenttimemillis  val blockstoreresult = receivedblockhandler.storeblock ( Blockid, receivedblock)   logdebug (S "pushed block  $blockId  in ${( System.currenttimemillis - time)} ms ")   val numRecords =  Blockstoreresult.numrecords  val blockinfo = receivedblockinfo (streamId,  Numrecords, metadataoption, blockstoreresult)   trackerendpoint.askwithretry[boolean] ( Addblock (Blockinfo))   logdebug (S "reported block&nbSP; $blockId ")} 

This method, the data is given to the Receiverblockhandler store, and the metadata is reported to the Receivertracker.

Receiverblockhandler is implemented in two ways:

private val receivedblockhandler: receivedblockhandler = {  if  ( Writeaheadlogutils.enablereceiverlog (env.conf))  {    if  ( Checkpointdiroption.isempty)  {      throw new sparkexception (          "Cannot enable receiver write-ahead log  without checkpoint directory set.  " +            "Please use streamingcontext.checkpoint ()  to set the  checkpoint directory.  " +          " see  Documentation for more details. ")     }    new writeaheadlogbasedblockhandler (Env.blockManager,  receiver.streamId,      receiver.storageLevel, env.conf,  Hadoopconf, cHeckpointdiroption.get)   } else {    new  Blockmanagerbasedblockhandler (Env.blockmanager, receiver.storagelevel)   }}

The data will eventually be handed to Blockmanager.

Blocksforpushing is defined as follows:

Private Val blockqueuesize = Conf.getint ("Spark.streaming.blockQueueSize", ten) private Val blocksforpushing = new Arrayblockingqueue[block] (blockqueuesize)

The blocksforpushing data is written by the Blockintervaltimer timer periodically to the data in the Blockgenerator Currentbuffer.

/** change the buffer to which single records are added to.  */private def updatecurrentbuffer (Time: long):  unit = {  try  {    var newBlock: Block = null     synchronized {      if  (Currentbuffer.nonempty)  {         val newBlockBuffer = currentBuffer         currentBuffer = new ArrayBuffer[Any]         val blockid = streamblockid (Receiverid, time - blockintervalms )         listener.ongenerateblock (Blockid)          newblock = new block (Blockid, newblockbuffer)        }   &nbsP;}     if  (newblock != null)  {       Blocksforpushing.put (Newblock)   // put is blocking when queue is  full    }  } catch {    case ie:  Interruptedexception =>      loginfo ("Block updating timer  thread was interrupted ")     case e: Exception =>       reporterror ("Error in block updating thread",  e)    }}

Let's go back and look at Supervisor's Startreceiver () method:

/** start receiver */def startreceiver (): unit = synchronized {   try {    if  (Onreceiverstart ())  {       Loginfo ("Starting receiver")       receiverState = Started       receiver.onstart ()       loginfo ("Called  Receiver onstart ")     } else {      //  The driver refused us      stop ("Registered unsuccessfully  because Driver refused to start receiver  " + streamid, none )     }  } catch {    case nonfatal (t)  = >      stop ("error starting receiver "  + streamId,  some (t))  &nbsP;}} 

Call receiver's OnStart method, we take socketreceiver as an example:

Def onStart () {//Start the thread that receives data over a connection new thread ("Socket Receiver") {Setdaemon (tr UE) override def run () {receive ()}}.start ()}

In this function, a new thread "Socket Receiver" is generated, and the thread initiates the call to Socketreceiver's receive () method

 def receive ()  {    var socket: Socket = null     try {      loginfo ("connecting to "  +  host +  ":"  + port)       socket = new  Socket (Host, port)       loginfo ("connected to "  + host  +  ":"  + port)       val iterator =  Bytestoobjects (Socket.getinputstream ())       while (!isStopped &&  iterator.hasnext)  {        store (iterator.next)        }      if  (!isstopped ())  {         restart ("Socket data stream had no more data")       }&nbsP;else {        loginfo ("stopped receiving")        }    } catch {      case  e: java.net.connectexception =>        restart (" error connecting to  " + host + ": " + port, e"        case nonfatal (e)  =>         Logwarning ("Error receiving data",  e)         restart (" Error receiving data ",  e)     } finally {       if  (socket != null)  {         Socket.close ()         loginfo ("closed socket to "   + host +  ":"  + p ort)       }    }  }} 

A socket object is built, and the data is continuously received from the InputStream, and the store method is called every time it is received.

def store (dataitem:t) {Supervisor.pushsingle (DataItem)}

The data is managed by Receiversupervisor and calls Supervisor.pushsingle to write the data.

def pushsingle (Data:any) {defaultblockgenerator.adddata (data)}

Defaultblockgenerator is defined as follows

Private Val defaultblockgenerator = Createblockgenerator (defaultblockgeneratorlistener) override Def Createblockgenerator (blockgeneratorlistener:blockgeneratorlistener): Blockgenerator = {//Cleanup BlockGenerators t Hat has already been stopped registeredblockgenerators--= registeredblockgenerators.filter{_.isstopped ()} Val NewBlo Ckgenerator = new Blockgenerator (Blockgeneratorlistener, Streamid, env.conf) registeredblockgenerators + = Newblockgenerator Newblockgenerator}

It is a blockgenerator, and the AddData function saves the data in the Currentbuffer object in Blockgenerator

/** * push a single data item into the buffer. */def  AddData (Data: any): unit = {  if  (state == active)  {     waittopush ()     synchronized {       if  (state == active)  {        currentbuffer + = data      } else {         throw new sparkexception (           "cannot  add data as blockgenerator has not been started or has  been stopped ")       }    }  } else  {    throw new sparkexception (       "cannot  add data aS blockgenerator has not been started or has been stopped ")    }}

As the data flows in, each 200ms writes the data in the Currentbuffer to the blocksforpushing queue, and then instantiates a currentbuffer. The blocksforpushing queue is written to Blockmanager every 10ms.

Note:

1. DT Big Data Dream Factory public number Dt_spark
2, the IMF 8 o'clock in the evening big data real combat YY Live channel number: 68917580
3, Sina Weibo: Http://www.weibo.com/ilovepains

This article is from the "Ding Dong" blog, please be sure to keep this source http://lqding.blog.51cto.com/9123978/1774426

10th lesson: Spark Streaming Source interpretation data constantly receiving full life cycle thorough research and thinking