Event Loop processor Eventloop__spark in Spark

Spark Event Loop Processor EventLoop implementation of the principle is not difficult, although the implementation of their own probably not too much, but feel that their realization in the expansion of convenience is certainly not as good as the spark EventLoop, so write a blog record.

The event loop processor, as its name suggests, is intended to handle events. In spark, there are event handlers for the internal anonymous implementation of Dagschedulereventprocessloop event handlers and Jobgenerator. The event Loop processor implementation principle is relatively simple, the bottom uses a thread as the event rationale thread, uses a linkedblockingdeque as the event buffer, the temporary store receives the event to the event processing thread processing. A thread's handling of an event is through a while dead loop, monitoring the message queue and getting and processing whenever there is a message in the message team. If the message queue is empty, the take is blocked here. The code implementation is as follows:

* * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license.
 The NOTICE file distributed with * is work for additional information regarding copyright. * The ASF licenses this file to you under the Apache License, Version 2.0 * (the "License");  You are not to use this file except in compliance with * the License. Obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * unless required by appli Cable or agreed to in writing, software * Distributed under the License was distributed on ' as is ' basis, * Withou
 T warranties or CONDITIONS of any KIND, either express OR implied.
 * The License for the specific language governing permissions and * limitations under the License. * * Package Org.apache.spark.util import java.util.concurrent. {blockingqueue, linkedblockingdeque} import java.util.concurrent.atomic.AtomicBoolean Import Scala.util.control.NonFatal Import Org.apachE.spark.internal.logging/** * A event loop to receive events from the caller and process all events in the event thre Ad.
  It * 'll start a exclusive event thread to process all events. * * Note:the event queue would grow indefinitely.
  So subclasses should make sure ' onreceive ' can * handle events in time to avoid the potential OOM. * <br><br> * Inside a thread responsible for processing messages * */Private[spark] abstract class Eventloop[e] (name:string) extends Logging {/**eventqueue is an event queue/private val Eventqueue:blockingqueue[e] = new Linkedblockingdeque[e] ()/** event loop processor start or end tag

    * Private Val stopped = new Atomicboolean (false) private Val eventthread = new Thread (name) {Setdaemon (True)
          Override Def run (): unit = {try {while (!stopped.get) {val event = Eventqueue.take () try {//todo If there is an event, call OnReceive for message handling OnReceive (event)} catch {case Non
     Fatal (e) => try {           OnError (e)} catch {case nonfatal (e) => logerror ("Unexpected error in" + Nam E, E)}} and catch {case Ie:interruptedexception =>//exit even if EV Entqueue is isn't empty case nonfatal (e) => logerror ("Unexpected error in" + Name, E)}} def
    Start (): unit = {if (stopped.get) {throw new IllegalStateException (name + "has already been stopped")} Call OnStart before starting the event thread to make sure it happens before OnReceive OnStart () eventthread  . Start ()} def Stop (): unit = {if (Stopped.compareandset (False, True)) {Eventthread.interrupt () var Onstopcalled = False try {Eventthread.join ()//Call OnStop after the event thread exits to make s Ure onreceive happens before onStop onstopcalled = True OnStop ()} catch {case Ie:interrup
     Tedexception =>     Thread.CurrentThread (). Interrupt () if (!onstopcalled) {//IE is thrown from ' Eventthread.join ( )`.
            Otherwise, we should not call ' onStop ' since//it ' s already called.
    OnStop ()}} else {//Keep quiet to allow calling ' stop ' multiple times. }/** * Put the event into the event queue.
    The event thread would process it later. */Def post (event:e): unit = {eventqueue.put (event)}/** * return if the event thread has already been s
    tarted but not yet stopped.  */def Isactive:boolean = eventthread.isalive/** * Invoked when ' start () ' is called but the event thread
    Starts. * Life Cycle Method */protected Def onStart (): unit = {}/** * invoked when ' stop () ' is called and the event thread Exi
    Ts. * Life Cycle Method */protected Def onStop (): unit = {}/** * invoked in the event thread when polling events from the
    Event queue. * * Note:shouLD avoid calling blocking actions in ' onreceive ', or the event thread would be blocked * and cannot process events in T Ime.
    If you are want to call some blocking actions, run them in * another thread. * <br> because different services are handled differently, they are defined as abstract methods, with specific subclasses implementing <br><br> * Lifecycle methods based on specific business * * Protected def onreceive (event : E): Unit/** * invoked if ' onreceive ' throws any non fatal error.
    Any non fatal error thrown from ' onError ' * would be ignored.
 * <br> due to different business processing methods, so defined as abstract method, have specific subclasses according to specific business implementation/protected Def onError (e:throwable): unit}

Implementation, the common business implementation is implemented in EventLoop, and different business methods are defined as abstract methods, which facilitates the realization of specific business in later specific business implementations.