Spark Message Queuing mechanism source learning

SOURCE Learning

The spark source comment has the following sentence:

Asynchronously passes sparklistenerevents to registered Sparklisteners

That is, all spark message sparklistenerevents are sent asynchronously to an already registered sparklisteners.
In Sparkcontext, the Livelistenerbus instance is created first, and the main functions of this class are as follows:

• Hold message queue, responsible for caching of messages
• Save a registered listener, responsible for the distribution of the message

The inheritance hierarchy for this class is as follows:

The listener list is stored in the Listenerbus class, where the Java Copyonwritearraylist class is used to store the listener in order to ensure the security of concurrent access. When changes are required to the listener linked list, the Copyonwritearraylist feature causes the entire list to be copied and then modified on the copied list. When you get an iterator to the list, you can guarantee the consistency of the data in the life cycle of the iterator.

private[spark] trait ListenerBus[L <: AnyRef, E] extends Logging {  // Marked `private[spark]` for access in tests.  privatenew CopyOnWriteArrayList[L]  /**   * Add a listener to listen events. This method is thread-safe and can be called in any thread.   */  final def addListener(listener: L) {    listeners.add(listener)  }  ...

Message Queuing is actually saved in the class Asynchronouslistenerbus:

  private val EVENT_QUEUE_CAPACITY = 10000  private val eventQueue = new LinkedBlockingQueue[E](EVENT_QUEUE_CAPACITY)

The length of the event queue is 10000, and when the number of cache events reaches the limit, the new event is discarded, and the specific discard handler is in the Livelistenerbus class:

Private[Spark] class livelistenerbusextends asynchronouslistenerbus[Sparklistener , sparklistenerevent] ("sparklistenerbus")  with sparklistenerbus {     PrivateVal logdroppedevent =NewAtomicboolean (false) Override Def ondropevent (event:sparklistenerevent): Unit = {if(Logdroppedevent.compareandset (false,true)) {//Only log the following message once to avoid duplicated annoying logs.LogError ("dropping sparklistenerevent because no remaining in event queue."+"This likely means one of the sparklisteners are too slow and cannot keep up with"+"The rate at which tasks is being started by the scheduler.")    }  }}

The above code can be seen, processing output error logs, and by using the variable logdroppedevent to ensure output only once.
Continue to focus on the class Asynchronouslistenerbus, which is the core of the message mechanism. Since it is a message queue, it involves the production and consumption of messages. First of all, to see how the message is consumed, The Asynchronouslistenerbus class creates a consumer thread listenerthread to get messages from the message queue and distribute them, following the implementation code:

PrivateVal Listenerthread =NewThread (name) {Setdaemon (true)Overridedef run (): Unit = Utils.tryorstopsparkcontext (sparkcontext) { while(true) {Eventlock.acquire () self.synchronized {processingevent =true}Try{valEvent= Eventqueue.pollif(Event==NULL) {//Get out of the that while loop and shutdown the daemon thread            if(!stopped.Get) {Throw NewIllegalStateException ("Polling ' null ' from EventQueue means"+"The listener bus has been stopped. So ' stopped ' must is true ")            }return} posttoall (Event)        }finally{self.synchronized {processingevent =false}        }      }    }  }

The whole idea is a typical producer consumer thought. To ensure concurrent access of the producer and consumer to the message queue, call Eventlock.acquire () to get the semaphore each time a message is needed. The value of the semaphore is the number of events in the current queue. If you get to the event normally, call Posttoall to distribute the event to all listener and continue to the next loop. If you get a null value, there are two scenarios:

1. The entire application normally ends, at which point the stopped value has been set to True
2. The system has an error and immediately terminates the operation

below to see the producer, the code is as follows:

  def post(event: E) {    if (stopped.get) {      // Drop further events to make `listenerThread` exit ASAP      logError(s"$name has already stopped! Dropping event $event")      return    }    val eventAdded = eventQueue.offer(event)    if (eventAdded) {      eventLock.release()    else {      onDropEvent(event)    }  }

This function is used to put the event into the message queue, and each time a successful event is placed, the Eventlock.release () is called to increase the semaphore value. For consumer threads to consume. If the queue is full, call ondropevent to handle the function, which is already listed above, and is not covered here.

The true message route is done by Sparklistenerbus's onpostevent function:

Override Def onpostevent (Listener:sparklistener, event:sparklistenerevent): Unit = {Event Match { Casestagesubmitted:sparklistenerstagesubmitted = listener.onstagesubmitted (stagesubmitted) Casestagecompleted:sparklistenerstagecompleted = listener.onstagecompleted (stagecompleted) CaseJobstart:sparklistenerjobstart = Listener.onjobstart (Jobstart) CaseJobend:sparklistenerjobend = Listener.onjobend (jobend) CaseTaskstart:sparklistenertaskstart = Listener.ontaskstart (Taskstart) CaseTaskgettingresult:sparklistenertaskgettingresult = Listener.ontaskgettingresult (TaskGettingResult) CaseTaskend:sparklistenertaskend = Listener.ontaskend (taskend) CaseEnvironmentupdate:sparklistenerenvironmentupdate = Listener.onenvironmentupdate (environmentUpdate) Caseblockmanageradded:sparklistenerblockmanageradded = listener.onblockmanageradded (blockManagerAdded) Caseblockmanagerremoved:sparklistenerblockmanagerremoved = listener.onblockmanagerremoved (blockManagerRemoved) CaseUnpersistrdd:sparklistenerunpersistrdd = Listener.onunpersistrdd (Unpersistrdd) CaseApplicationstart:sparklistenerapplicationstart = Listener.onapplicationstart (Applicationstart) CaseApplicationend:sparklistenerapplicationend = Listener.onapplicationend (applicationend) CaseMetricsupdate:sparklistenerexecutormetricsupdate = Listener.onexecutormetricsupdate (metricsUpdate) Caseexecutoradded:sparklistenerexecutoradded = listener.onexecutoradded (executoradded) Caseexecutorremoved:sparklistenerexecutorremoved = listener.onexecutorremoved (executorremoved) Caseblockupdated:sparklistenerblockupdated = listener.onblockupdated (blockupdated) CaseLogstart:sparklistenerlogstart =//Ignore event log metadata}  }

The above code is very intuitive, depending on the message type, call listener corresponding method for processing.
Here's a look at the definition of listener, where all the listener are mixed with sparklistener traits. This trait defines a handler function for all messages, all of which are empty:

@DeveloperApitrait SparkListener {  /**   * Called when a stage completes successfully or fails, with information on the completed stage.   */  def onStageCompleted(stageCompleted: SparkListenerStageCompleted) { }  /**   * Called when a stage is submitted   */  def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted) { }  ......

For a particular listener, after mixing the sparklistener trait, only the corresponding handler function can be rewritten. To facilitate pattern matching during message routing, all specific listener classes are defined as sample classes.

For events in Spark, thinking is similar to listener, except that the qualities of the mix are different, and the characteristics of the events are sparklistenerevent.

Message Queuing Setup and delivery process description:

The specific message sending process is as follows:

In Sparkcontext, you will

1. Create a member variable for the Livelistenerbus class type Listenerbus
2. Create various listener and add to Listenerbus
3. Post some events into the Listenerbus
4. Call Listenerbus.start () to start the event handler

One thing to note here is that before the Listenerbus.start () call, a post message can be sent to it, the messages are cached, and the consumer thread distributes the cached messages after the start function is called. Listenerbus.start () is called in the Setupandstartlistenerbus function in Sparkcontext, and the following is a look at the implementation of the function:

  Private defSetupandstartlistenerbus (): Unit = {//Use reflection to instantiate listeners specified via ' Spark.extralisteners '    Try{ValListenerclassnames:seq[string] = Conf.get ("Spark.extralisteners",""). Split (', '). Map (_.trim). Filter (_! ="") for(ClassName <-Listenerclassnames) {//Use reflection to find the right constructor        Valconstructors = {ValListenerclass = Utils.classforname (className) Listenerclass.getconstructors.asinstanceof[array[constructor[_ < : Sparklistener]]}Valconstructortakingsparkconf = Constructors.find {c = = C.getparametertypes.sameelements (Array (classOf[SparkConf ]))        }Lazy ValZeroargumentconstructor = constructors.find {c = c.getparametertypes.isempty}ValListener:sparklistener = {if(constructortakingsparkconf.isdefined) {constructorTakingSparkConf.get.newInstance (conf)}Else if(zeroargumentconstructor.isdefined) {zeroArgumentConstructor.get.newInstance ()}Else{Throw NewSparkexception (S"$className did not have a zero-argument constructor or a"+"Single-argument constructor that accepts sparkconf. Note:if the class is "+"defined inside of another Scala class, then it constructors may accept"+"Implicit parameter that references the enclosing class; In this case, you must "+"Define the listener as a top-level class in order to prevent this extra"+"parameter from breaking Spark's ability to find a valid constructor.")}} listenerbus.addlistener (Listener) Loginfo (s"Registered listener $className")      }    }Catch{ CaseE:exception =Try{Stop ()}finally{Throw NewSparkexception (S"Exception when registering Sparklistener", e)}} listenerbus.start ( This) _listenerbusstarted =true}

This code first uses the reflection mechanism to handle the Spark.extralisteners setting, and there is an explanation for that setting in spark Doc:

A comma-separated List of classes that implement Sparklistener; When initializing Sparkcontext, instances of these classes would be created and registered with Spark ' s listener bus. If A class has a single-argument constructor that accepts a sparkconf, that constructor would be called; Otherwise, a zero-argument constructor would be called. If no valid constructor can be found, the Sparkcontext creation would fail with an exception.

The listener of this setting is created when the Sparkcontext is initialized, but is required for listener constructors:

• Has a single-argument constructor and the argument is a sparkconf type, the constructor is called
• Otherwise, if there is no parameter constructor, it will be called
• If no constructor is defined, the program ends unexpectedly

When Extralisteners is constructed and registered, Listenerbus.start is called:

def start(sc: SparkContext) {    if (started.compareAndSet(falsetrue)) {      sparkContext = sc      listenerThread.start()    else {      thrownew IllegalStateException(s"$name already started!")    }  }

At the same time, start the consumer thread listenerthread and start the message routing.

When the program finishes running, the Stop function is called:

def stop () {if(!started.get ()) {Throw NewIllegalStateException (S"attempted to stop $name that have not yet started!")    }if(Stopped.compareandset (false,true)) {//Call Eventlock.release () so that Listenerthread would poll 'NULL' From 'EventQueue` andKnow//`Stop` isCalled. Eventlock.release () Listenerthread.join ()}Else{//Keep Quiet}}

As you can see here, Eventlock.release () is called in the Stop function to increase the semaphore value. However, no new messages are added to the message queue, which results in the end of the Listenerthread thread by returning a null value when the consumer thread Listenerthread read the queue.

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Spark Message Queuing mechanism source learning