Spark Growth Path (5)-Message Queuing

Reference article: Spark distributed Message delivery process
Listener mode
Volatile

Because before by this message queue pit (stage tamping reason analysis), so now research source, first from it, answer me for so long doubts. Inheritance Relationship

Listenerbus->sparklistenerbus->livelistenerbus. The original base class is Listenerbus. The design pattern used is listener mode. Listenerbus

The private trait in the spark package, inherited from logging (convenient output log), to familiarize itself with its member variables listenersplustimers member variables

Class private variable, copyonwritearraylist is a thread-safe list that can be treated as a ArrayList (+ thread-safe). The object being stored is a two-tuple, the first element of the two-tuple is a reference object, the second element is an option object, the listener's timer is obtained, and none is available.

Private[this] Val listenersplustimers = new copyonwritearraylist[(L, Option[timer])

Listeners Method

Convert Copyonwritearraylist to a collection of Java, this should be util. List.

Private[spark] def listeners = ListenersPlusTimers.asScala.map (_._1). Asjava

AddListener Method

The method is defined with final, which means that it cannot be overridden by inheritance. This method is one of the standard methods for adding observers, listener mode.

Final def AddListener (listener:l): Unit = {
    Listenersplustimers.add ((Listener, Gettimer (listener))
  }

RemoveListener Method

Listener Mode standard Formula Two, remove the listener

Final def RemoveListener (listener:l): Unit = {
    listenersPlusTimers.asScala.find (_._1 eq listener). foreach { Listenerandtimer =
      listenersplustimers.remove (listenerandtimer)
    }
  }

Gettimer Method

Get the timer in the listener

Protected def gettimer (listener:l): option[timer] = None

Posttoall Method

Sends an event to each listener.

def posttoall (event:e): Unit = {
    //javaconverters can create a jiterablewrapper if we use Asscala.
    However, this method would be called frequently. To avoid the wrapper cost, here we use
    //Java Iterator directly.
    Val iter = Listenersplustimers.iterator while
    (Iter.hasnext) {
      val listenerandmaybetimer = Iter.next ()
      Val Listener = listenerandmaybetimer._1
      val maybetimer = listenerandmaybetimer._2
      val maybetimercontext = if ( maybetimer.isdefined) {
        maybeTimer.get.time ()
      } else {
        null
      }
      try {
        dopostevent ( Listener, event)
      } catch {case
        nonfatal (e) =
          LogError (S "Listener ${utils.getformattedclassname ( Listener)} threw an exception ", E)
      } finally {
        if (maybetimercontext! = null) {
          maybetimercontext.stop () c21/>}}}}
  

Findlistenersbyclass Method

The corresponding listener list is reflected according to the class name.

Private[spark] def findlistenersbyclass[t <: L:classtag] (): seq[t] = {
    val c = Implicitly[classtag[t]].runtimecla SS
    Listeners.asScala.filter (_.getclass = = c). Map (_.asinstanceof[t]). Toseq
  }

Dopostevent Method

Sends an event to the specified listener, which is specifically implemented by subclasses.

Protected Def dopostevent (Listener:l, event:e): Unit

Sparklistenerbus

Previously said Dopostevent This method, by sub-class to achieve. OK, let's see how the Sparklistenerbus came true.

Sparklistenerbus is very embarrassed no skin only this one method, and nothing else, the class has put listener and event specific points to sparklistenerinterface and sparklistenerevent. This method uses pattern matching to specifically invoke the method of sending events from different listeners.

protected override Def dopostevent (Listener:sparklistenerinterface, event:sparklistenerevent): Unit = { Event Match {case stagesubmitted:sparklistenerstagesubmitted = listener.onstagesubmitted (Stagesubmitt ed) Case stagecompleted:sparklistenerstagecompleted = listener.onstagecompleted (stagecompleted) c ASE Jobstart:sparklistenerjobstart = Listener.onjobstart (jobstart) Case Jobend:sparklistenerjobend =&G
        T
      Listener.onjobend (jobend) Case taskstart:sparklistenertaskstart = Listener.ontaskstart (TaskStart)
      Case Taskgettingresult:sparklistenertaskgettingresult = Listener.ontaskgettingresult (TaskGettingResult) Case taskend:sparklistenertaskend = listener.ontaskend (taskend) Case Environmentupdate:sparklistener Environmentupdate = listener.onenvironmentupdate (environmentupdate) Case Blockmanageradded:sparklistene RblockmanagerAdded = listener.onblockmanageradded (blockmanageradded) Case Blockmanagerremoved:sparklistenerblockmana gerremoved = listener.onblockmanagerremoved (blockmanagerremoved) Case Unpersistrdd:sparklistenerunpersi 
        Strdd = Listener.onunpersistrdd (unpersistrdd) Case Applicationstart:sparklistenerapplicationstart =
        Listener.onapplicationstart (applicationstart) Case applicationend:sparklistenerapplicationend = Listener.onapplicationend (applicationend) Case metricsupdate:sparklistenerexecutormetricsupdate = Lis Tener.onexecutormetricsupdate (metricsupdate) Case executoradded:sparklistenerexecutoradded = LISTENER.O nexecutoradded (executoradded) Case executorremoved:sparklistenerexecutorremoved = Listener.onexecutorre Moved (executorremoved) Case executorblacklisted:sparklistenerexecutorblacklisted = Listener.onexecutorb Lacklisted (ExecutorblackListed) Case executorunblacklisted:sparklistenerexecutorunblacklisted = Listener.onexecutorunblackliste D (executorunblacklisted) Case nodeblacklisted:sparklistenernodeblacklisted = listener.onnodeblacklisted (nodeblacklisted) Case nodeunblacklisted:sparklistenernodeunblacklisted = listener.onnodeunblacklisted (
      nodeunblacklisted) Case blockupdated:sparklistenerblockupdated = listener.onblockupdated (blockUpdated) Case _ = = Listener.onotherevent (event)}}

I have time to Sparklistenerinterface, sparklistenerevent these two subsystems logical rationale. Livelistenerbus

The class has a companion object and a subordinate object, Livelistenerbusmetrics. Next hit explains EventQueue member variables

Message Queuing, which I write at the core of this article, explains why you should limit the size of this queue, because if you add more speed than you consume, it can cause oom. If you do not set the Spark.scheduler.listenerbus.eventqueue.size parameter, the default is 10000.

Cap the capacity of the event queue so we get a explicit error (rather than
  //an OOM exception) if it ' s perpetual Ly being added to more quickly than it ' s being drained.
  Private Val eventqueue =
    new Linkedblockingqueue[sparklistenerevent] (Conf.get (listener_bus_event_queue_ capacity))

Metrics member Variables

Private[spark] Val metrics = new Livelistenerbusmetrics (conf, eventqueue)

The object that holds each configuration. Listenerthread member Variables

Private Val listenerthread = new Thread (name) {Setdaemon (true) override Def run (): Unit = Utils.tryorstopsparkcon Text (sparkcontext) {LiveListenerBus.withinListenerThread.withValue (true) {val timer = metrics.eventprocess Ingtime while (true) {Eventlock.acquire () self.synchronized {processingevent = t Rue} try {val event = Eventqueue.poll if (event = = null) {// Get out of the while loop and shutdown the daemon thread if (!stopped.get) {throw new Illeg Alstateexception ("Polling ' null ' from eventqueue means" + "the listener bus has been stopped. 
            So ' stopped ' must is true ")} return} val Timercontext = Timer.time ()
          try {Posttoall (event)} finally {Timercontext.stop ()}
           } finally { self.synchronized {processingevent = False}}}}} 

The thread object is first set to the daemon. Then pass the code block to the Utils.tryorstopsparkcontext method to execute, this method has the function of catch exception, found the exception to stop Sparkcontext. Then, using LiveListenerBus.withinListenerThread.withValue (TRUE), the variable is set to True after the subsequent code block is executed, and the original value is set back after execution is complete. Then go to the dead loop, set Processingevent to true before and after executing the statement, indicating that the actual process of handling the event is set to False to identify the idle state after processing. The message is then poll from the message queue and sent. The timer's role makes the calculation time-consuming.

Overall, this method is an asynchronous execution of threading methods, with thread-safe, locking mechanisms. Specifically for what to add to these things, it is necessary to study what the caller is aware of. Start Method

Ingress method, caller is sparkcontext:

is actually called execute "listenerthread thread

def start (Sc:sparkcontext, metricssystem:metricssystem): Unit = {
    if (Started.compareandset (False, True)) {
      Sparkcontext = SC
      metricssystem.registersource (metrics)
      Listenerthread.start ()
    } else {
      throw new IllegalStateException (S "$name already started!")
    }
  }

Post Method

The

Adds an event to the queue. Val eventadded = Eventqueue.offer (event) This line of code adds the element to the queue, returns True if the add succeeds, and returns false if the queue is full. If the queue is full, then if the queue is more than the length of the new message will be discarded, it is very embarrassing.

Def post (event:sparklistenerevent): Unit = {if (stopped.get) {//Drop further events to make ' Listenerthread ' Exit ASAP LogError (S "$name has already stopped!
    Dropping event $event ") return} metrics.numEventsPosted.inc () Val eventadded = Eventqueue.offer (event) if (eventadded) {eventlock.release ()} else {ondropevent (event)} val droppedevents = Dropp  Edeventscounter.get if (droppedevents > 0) {//Don ' t log too frequently if (System.currenttimemillis () -Lastreporttimestamp >= * +) {//There may be multiple threads trying to decrease droppedeventscounter
        .
        Use "Compareandset" to make sure only one thread can win. And if another thread is increasing droppedeventscounter, "Compareandset" would fail and//then that thread wil
        L Update it. if (Droppedeventscounter.compareandset (droppedevents, 0)) {val Prevlastreporttimestamp = LastreporttimEstamp Lastreporttimestamp = System.currenttimemillis () logwarning (S "Dropped $droppedEvents sparkliste
 Nerevents since "+ New Java.util.Date (Prevlastreporttimestamp)}}}}}

If the addition fails to call Ondropevent, the relevant counter is added 1, and output log, output only once.

def ondropevent (event:sparklistenerevent): Unit = {
    metrics.numDroppedEvents.inc ()
    Droppedeventscounter.incrementandget ()
    if (Logdroppedevent.compareandset (False, True)) {
      //only log the Following message once to avoid duplicated annoying logs.
      LogError ("dropping sparklistenerevent because no remaining in event queue." + "This likely means one of the
        Spar Klisteners is too slow and cannot keep up with "+" the rate at
        which tasks was being started by the scheduler. ")
    }
  }

Because a log message with a full message queue is output only once at a time, it is reset after more than one minute:

Val droppedevents = Droppedeventscounter.get
    if (droppedevents > 0) {
      //Don ' t log too frequently
      if (syste M.currenttimemillis ()-lastreporttimestamp >= * +) {
        //There may be multiple threads trying to decrease DRO Ppedeventscounter.
        Use "Compareandset" to make sure only one thread can win.
        And if another thread is increasing droppedeventscounter, "Compareandset" would fail and
        //then that thread would up Date it.
        if (Droppedeventscounter.compareandset (droppedevents, 0)) {
          val prevlastreporttimestamp = Lastreporttimestamp
          Lastreporttimestamp = System.currenttimemillis ()
          logwarning (S "Dropped $droppedEvents sparklistenerevents Since "+
            new Java.util.Date (Prevlastreporttimestamp))}}}
    

Let's see who called this post method to add events to the queue.

With Dagscheduler,sparkcontext, I will detail this dagscheduler in the next article in the task call. In general, we are responsible for adding events in these two classes, but we are a distributed system, which has the concept of master and worker inside, and executes the architecture diagram:

As can be seen from the figure, Dagscheduler is present in the Sparkcontext, subordinate to master, that the worker and master directly through the NIO message delivery, Master received the information and resolved, Delegate Dagscheduler to commit the event of the response. Summary

The research of message queue is over, found that there are many things to learn, now just learning spark-core things, I really do spark-streaming still waiting for me. Time is not enough AH ~