Spark Core Source Analysis 4 worker initiation process

SOURCE location:org.apache.spark.deploy. worker. Worker.scala

First, look at the worker's main method, similar to master, create a sparkconf, parse the parameters, and construct the worker object and create the Actorref for external or own information interaction. Here the Masters parameter can be set to multiple

def main (argstrings:array[string]) {  signallogger.register (log)  val conf = new sparkconf  val args = new work Erarguments (argstrings, conf)  Val (Actorsystem, _) = Startsystemandactor (Args.host, Args.port, Args.webuiport, Args.cores,    args.memory, Args.masters, Args.workdir)  actorsystem.awaittermination ()}

After the procedure, it is also the Prestart method to execute Akka first.

override  Def Prestart () {assert (!registered) loginfo ("Starting Spark worker%s:%d with%d cores,%s RAM". Format (host, Port, Cores, utils.megabytestostring (memory))) Loginfo (S "Running Spark version ${org.apache.spark.spark_version}") Loginfo ("Spark Home:" + sparkhome) Createworkdir ()//Create worker internal working directory//Subscription Akka life cycle event context.system.eventStream.subscribe (self  , classof[remotinglifecycleevent])//Whether an additional shuffle service is initiated to ensure that the shuffle file read and written by Executor is saved after the executor exits, and can be shuffleservice.startifenabled () WebUi = new Workerwebui (this, Workdir, Webuiport) Webui.bind () Registerwithmaster ()//most Important action, see below Metricssystem.registersource (Workersource) Metricssystem.start ()//Attach The worker metrics servlet handle  R to the Web UI after the metrics system is started. MetricsSystem.getServletHandlers.foreach (Webui.attachhandler)} 
 Register yourself with master 

Private Def registerwithmaster () {  //disassociatedevent May is triggered multiple times, so don ' t attempt registratio n  //If there is outstanding registration attempts scheduled.  Registrationretrytimer Match {Case None = +      registered = False      //here to all master Actorref sends the Registerworker message, the last few sections have said that master received the message, if the successful processing will be feedback Registeredworker message, unsuccessful will send registerworkerfailed message      Tryregisterallmasters ()      connectionattemptcount = 0      //Here After a certain time will enter the Reregisterwithmaster, which will determine whether it is registered, If no registration information is sent again. The status of this registration is returned by master feedback      Registrationretrytimer = Some {        Context.system.scheduler.schedule (Initial_ Registration_retry_interval,          initial_registration_retry_interval, Self, reregisterwithmaster)      }    Case Some (_) =      Loginfo ("not spawning another attempt to register with the master, since there was an" +        "att Empt scheduled already. ")  }}
See if the worker receives master'sWhat will the Registeredworker message do? To say this, the worker does not know which one is the master and which is the standby when registering, so it sends the registration information to all the configured master. After the master has received the worker's registration information, only the master will feedback, and bring his own masterurl information, the worker as the main master of the Actorref for real information interaction

The worker wants to maintain the connection with the master through the heartbeat, so after the successful registration, there is a connected flag is connected properly, Changemaster method Internal Settings connected = True

<pre name= "code" class= "Java" >case registeredworker (MasterUrl, masterwebuiurl) =  Loginfo (" Successfully registered with Master "+ masterurl)  registered = TRUE//status set to registered, otherwise, after a certain period of time, Will initiate reregisterwithmaster and re-register  changemaster (MasterUrl, Masterwebuiurl)//This is to save the master master's information/  /after successful registration, Just turn on the timer to send the heartbeat to master  context.system.scheduler.schedule (0 millis, Heartbeat_millis millis, self, sendheartbeat)  //Timer cleanup Workdir is not updated for a long time and the app is not in the execution state of the directory  if (cleanup_enabled) {    Loginfo (S "Worker CLEANUP ENABLED; Application directories'll be deleted in: $workDir ")    Context.system.scheduler.schedule (cleanup_interval_millis Millis,      Cleanup_interval_millis millis, Self, workdircleanup)  }
If you receive registerworkerfailed message, Exit

Here's how Master takes care of the worker's heartbeat

Because the worker registers, master has already deposited the Workerid in Idtoworker, so here goes some branch. Very simple, just update the worker with a timestamp. Here it is necessary to explain the None branch, after the registration message arrives, in the master's Idtoworker and workers will be saved, but when Master detects the worker timeout, the worker is removed from the Idtoworker, This new task will not be able to select the worker, but does not delete the workers. It is only after a long interval that the workers does not have a heartbeat that the worker really can no longer work, and then removes it from the workers. The None branch here is the response timeout after which the heartbeat continues to come up and sends a re-registration message to the worker  reconnectworker

Case Heartbeat (Workerid) = {  Idtoworker.get (Workerid) match {case    Some (workerinfo) =      Workerinfo.lastheartbeat = System.currenttimemillis () case    None =      if (Workers.map (_.id). Contains ( Workerid) {        logwarning (S "Got heartbeat from unregistered worker $workerId." +          "asking it to re-register.")        Sender! Reconnectworker (MasterUrl)      } else {        logwarning (S "Got heartbeat from unregistered worker $workerId." +          " This worker is never registered, so ignoring the heartbeat. ")}}  

At this point, the worker initiation process and the unsolicited message is finished, the rest is passively received and processed by the process, after the combination of specific job description ...

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Spark Core Source Analysis 4 worker initiation process