Master ha thorough decryption (dt Big Data DreamWorks)

Content:

1, Master ha parsing;

2, Master ha of four ways;

3, the internal working mechanism of Master ha;

4, the Master Ha source code decryption;

This talk main source angle Analysis Master HA, because in the production environment must do

==========master ha parsing ============

Spark is the structure of Master-slave

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

Now the industry is 1 master active,2 more than standby

If HA is active, it will continue to run on the last run.

Drvier commit the program, request resources, and interact with master

Zookeeper work uses the leader mechanism, it provides services externally, the others are follower

Zookeeper retains the information of the cluster: Worker, Driver, application information, will be persisted to the zookeeper, when the switch will only affect the new job submission.

Since each job runs, as long as the cluster requests to the resources, and master does not matter, then the interaction between driver and executor, said in 27. This will only affect the new job submission, without affecting the operation of the existing job.

Summary:

1, the production environment is generally used zookeeper do ha, and recommended for 3 Master,zookeeper will be automated management masters switch;

2, the use of zookeeper do ha, zookeeper will save the entire spark cluster runtime metadata: All workers, Drivers, applications, executors, etc.;

3, zookeeper in the current active level of master failure, will be selected from the standby master as active master, but note: After being elected to become a real active Master needs to obtain the metadata information of the current running state of the cluster from zookeeper and restore it.

4, in the master switch process, all already running programs are normal operation! Because the spark application has obtained the compute resources through the cluster Manager before running, there is no relationship between the scheduling and processing of the job itself and master .

5, the only effect in the master switching process is that the new job cannot be submitted: On the one hand, the new application cannot be submitted to the cluster, because only active master can receive the new program's submission request, on the other hand, It is also not possible for a program that has already run to trigger a new job submission request because of an action action;

Experience: Yarn mode is about 30% less than standalone mode performance

Four ways to ==========master ha ============

1, Masterha Four ways are: ZOOKEEPER, FILESYSTEM, custom (customized), none (do not do ha, download direct use of spark);

2, the need to explain is:

1) Zookeeper is automatically managed master;

2) FileSystem the way to manually restart the machine after master failure, the machine will immediately become the active level of master to provide services (accept requests submitted by the application, accept the new job run request);

3) Custom mode allows users to customize the implementation of the MASTERHA, which is particularly useful for advanced users;

4) NONE, this is the default, when we download the installation of the spark cluster, this is the way, this way does not persist the cluster of data, master boot immediately after the management of the cluster;

Master in OnStart

Val Serializer =Newjavaserializer (conf)
Val(Persistenceengine_, LEADERELECTIONAGENT_) =Recovery_modeMatch{
Case"ZOOKEEPER"=
Loginfo ("Persisting recovery state to ZooKeeper")
ValZkfactory =
NewZookeeperrecoverymodefactory (conf, Serializer)
(Zkfactory.createpersistenceengine (), Zkfactory.createleaderelectionagent ( This)//Data persistence engine and leader elections
Case"FILESYSTEM"=
ValFsfactory =
NewFilesystemrecoverymodefactory (conf, Serializer)
(Fsfactory.createpersistenceengine (), Fsfactory.createleaderelectionagent ( This))
Case"CUSTOM"=
ValClazz = Utils.Classforname(Conf.get ("Spark.deploy.recoveryMode.factory"))
ValFactory = Clazz.getconstructor (classof[Sparkconf], classof[Serializer])
. newinstance (Conf, Serializer)
. Asinstanceof[standalonerecoverymodefactory]
(Factory.createpersistenceengine (), Factory.createleaderelectionagent ( This))
Case_ =
(NewBlackholepersistenceengine (), NewMonarchyleaderagent ( This))
}
Persistenceengine= Persistenceengine_
leaderelectionagent= Leaderelectionagent_

4, Persistengine has a crucial method persist to achieve data persistence, readpersisteddata to reply to the metadata in the cluster;

/**
* Returns the persisted data sorted by their respective IDs (which implies that they ' re
* Sorted by time of creation).
 */
Final defReadpersisteddata(
RPCENV:RPCENV): (Seq[ApplicationInfo], Seq[Driverinfo], Seq[Workerinfo]) = {
rpcenv.deserialize {() =
(Read[applicationinfo] ("App_"), Read[driverinfo] ("Driver_"), Read[workerinfo] ("Worker_"))
}
}

5, filesystem and none are used monarchyleaderagent way to complete the leader election, the actual realization is to direct the incoming master as leader

def createleaderelectionagent (master:leaderelectable): Leaderelectionagent = {
New Monarchyleaderagent (Master)
}

/** Single-node Implementation of leaderelectionagent--we ' re initially and always the leader. */
Private [Spark] class Monarchyleaderagent (val masterinstance:leaderelectable)
extends leaderelectionagent {
Masterinstance.electedleader ()
}

6, none does not need to persist, why wrote Blockholepersistenceengine, inside what did not realize? Code structure is unified, and easy to expand;

Private[Master]classBlackholepersistenceengineextendsPersistenceengine {

Override Defpersist(Name:String, Obj:object):Unit= {}

Override Defunpersist(Name:String):Unit= {}

Override DefRead[T: Classtag] (name:String):Seq[T] =Nil

}

Internal working mechanism of ==========master ha (main zookeeper) ============

1, zookeeper automatically from the standby master inside selected as the master of leader;

2, use Zookeeprpersistengine to read the status data of the cluster workers, Drivers, applications, executors and other information;

3, determine whether the metadata information is empty content;

4, the workers, Drivers, applications, executors and other information obtained through the Zookeeper persistence engine are re-registered to the memory of master cache;

5. Verify the consistency of the information obtained and the status of the currently running cluster;

6, the status of applications and workers is identified as unkown, then the driver in application and the worker is now leader mode of the master's address information;

7, when drivers and workers received the new master address information, will respond to change information;

8, Master received from Drviers and Workers response information, will use a key method Completerecovery, to unresponsive applications (Drivers), Workers (executors) Processing, the state of master becomes recoverystate. ALIVE , so you can start serving

Private DefCompleterecovery() {
//ensure "only-once" recovery semantics using a short synchronization period.
  if( State! = Recoverystate.RECOVERING) {return}
State= Recoverystate.Completing_recovery

  /Kill off any workers and apps this didn ' t respond to us.
  Workers. Filter (_. State= = Workerstate.UNKNOWN). foreach (Removeworker)
Apps. Filter (_. State= = ApplicationState.UNKNOWN). foreach (Finishapplication)

//Reschedule drivers which were not claimed by any workers
  Drivers. Filter (_.worker. IsEmpty). foreach {d = =
Logwarning (S "Driver${D.id}Was wasn't found after master recovery ")
if(d.desc.supervise) {
Logwarning (S "re-launching${D.id}")
Relaunchdriver (d)
}Else{
Removedriver (d.id, Driverstate.ERROR, None)
Logwarning (S " did not re-launch${D.id}because it is not supervised ")
}
}

State= Recoverystate.ALIVE
 Schedule ()
Loginfo ("Recovery complete-resuming operations!")
}

D.desc.supervise This method restarts after Drvier failure drvier

9,(Critical step) at this point, master calls its own scheduler method for the waiting applications and drviers resource scheduling !!!

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

This article from "a Flower proud Cold" blog, declined reprint!

Master ha thorough decryption (dt Big Data DreamWorks)