Spark Technical Insider: Client,master and worker communication Source analysis

Spark's cluster manager can have several deployment modes:

Standlone

Mesos

YARN

EC2

Local

After submitting the computing task to the cluster, the operation model of the system is submitted by the driver program definition Sparkcontext to the app master, and the computing resource is dispatched by the app Master, and finally the calculation is completed. Specific elaboration can read "Spark: Big Data of the electric flower flint!" 》。

So in standalone mode, how do client,master and worker communicate, register and open services?

1. Rpc-akka between node

Communication between modules has a lot of mature implementation, now many mature framework has already let us get rid of the original socket programming. The simple classification can be summed up as message-based and resource-sharing based synchronization mechanism.

The application of message-based transmission mechanism is more widely used in messages Queue. Message Queue, an application's method of communicating with applications. Applications communicate by reading and writing messages to and from queues (for application data) without having to link them with a private connection. Messaging refers to the fact that programs communicate with each other by sending data in a message, rather than by directly calling each other, a direct call is usually a technique for a remote procedure call. Queuing refers to the application's communication through queues. The use of queues eliminates the need for both receiving and sending applications to execute concurrently. Among the more mature MQ products are IBM WEBSPHERE MQ and RABBITMQ (AMQP's Open source implementation, now maintained by pivotal).

And what has to be mentioned is ZEROMQ, a socket-based programming framework dedicated to getting into the Linux kernel. The official saying: "Zeromq is a simple and easy to use the transport layer, like the framework of a socket library, which makes the socket programming simpler, concise and higher performance." is a message processing queue library that can flex across multiple threads, cores, and host boxes. ZMQ's clear goal is to "become part of the standard network protocol stack and then enter the Linux kernel."

Spark's choice of communication among many modules is the Akka of Scala native support, a library written in Scala to simplify the writing of fault-tolerant, highly scalable Java and Scala Actor model applications. Akka has the following 5 features:

Easy to build parallel and distributed applications (simple concurrency & Distribution): Akka used asynchronous communication and distributed architecture in design, and abstracted the upper layer, such as actors, Futures, STM, etc.

Reliability (resilient by Design): The system has self-healing capabilities and is monitored locally/remotely.

High performance: 50,000,000 messages can be sent per second in a single machine. Memory footprint is small, 2,500,000 actors can be saved in 1GB memory.

Elasticity, no Center (elastic-decentralized): Adaptive responsible for equalization, routing, partitioning, configuration

Extensible (extensible): can be extended using the Akka expansion pack.

The Client,master and the worker in the spark are actually a actor, taking the client:

Import akka.actor._ Import akka.pattern.ask import akka.remote. {associationerrorevent, disassociatedevent, remotinglifecycleevent} private class Clientactor (driverargs:cli Entarguments, conf:sparkconf) extends Actor with Logging {var masteractor:actorselection = _ val Timeout = Akkautils.asktimeout (conf) override Def prestart () = {Masteractor = Context.actorselection (Master.     
        
    Toakkaurl (Driverargs.master)) Context.system.eventStream.subscribe (self, classof[remotinglifecycleevent]) println (S "Sending ${driverargs.cmd} command to ${driverargs.master}") Driverargs.cmd ma tch {case "launch" => ... masteractor!     
        Requestsubmitdriver (driverdescription) case "Kill" => val DriverID = Driverargs.driverid Val killfuture = masteractor! Requestkilldriver (DriverID)}} override DefReceive = {case Submitdriverresponse (success, DriverID, message) => println (message)  if (success) Pollandreportstatus (driverid.get) Else System.exit ( -1) Case Killdriverresponse (DriverID, Success, message) => println (Message) if (success) Pollandreportstatus (DriverID) Else System.exit ( -1 Case Disassociatedevent (_, Remoteaddress, _) => println (S "Error Connecting to master ${d    
      Riverargs.master} ($remoteAddress), exiting. ") System.exit ( -1) case associationerrorevent (Cause, _, remoteaddress, _) => println (S "Error Co    
      Nnecting to Master ${driverargs.master} ($remoteAddress), exiting. ") println (S "Cause is: $cause") System.exit ( -1)}}/** * executable utility for Starti   
 Ng and terminating drivers inside of a standalone cluster. */Object Client {def main (args:array[string]) {PrinTLN ("Warning:this client is deprecated and would be removed in a future version of Spark.")    
    println ("use./bin/spark-submit with \"--master spark://host:port\ ") val conf = new sparkconf ()    
      Val Driverargs = new Clientarguments (args) if (!driverargs.loglevel.isgreaterorequal (Level.warn)) { Conf.set ("Spark.akka.logLifecycleEvents", "true")} conf.set ("Spark.akka.askTimeout", "ten") c Onf.set ("Akka.loglevel", DriverArgs.logLevel.toString.replace ("WARN", "WARNING")) Logger.getRootLogger.setLevel (d Riverargs.loglevel)//Todo:see If we can initialize Akka so return messages are sent back using the SAM e TCP//flow.    
    Else, this (sadly) requires the driverclient is routable from the Master. Val (Actorsystem, _) = Akkautils.createactorsystem ("Driverclient", Utils.localhostname (), 0, Conf, new security Manager (conf)) Actorsystem.actorof (Props (Classof[clientactor], Driverargs, conf) actorsystem.awaittermination ()}} 

The meaning of line 19th is to submit a driver request to master,

Masteractor! Requestsubmitdriver (driverdescription)

and master will process the request in receive. Of course, 27 to 44 lines are processed by the client actor received messages.

Can be seen, through Akka, can be very simple and efficient processing of communication between modules, which can be said to be a major feature of Spark RPC.