Spark Set-PLATE: 009~spark Streaming source interpretation of receiver in the driver of the subtle realization of the full life cycle of thorough research and thinking

The content of this lecture:

A. Receiver start-up approach envisaged
B. Receiver start source thorough analysis

Note: This lecture is based on the spark 1.6.1 version (the latest version of Spark in May 2016).

Previous section Review

In the previous lecture, we gave you a specific analysis of the RDD physical generation and logic generation process, thoroughly understand the relationship between the Dstream and the RDD, and its other related classes of specific dependencies and other information:

A. dstream is a template for the RDD, and its internal Generatedrdds holds an instance of the Rdd object generated per batchduration time. The dependency of Dstream constitutes the rdd dependency, which is calculated from the backward forward, as long as the last Dstream is computed.

B. Jobgenerator every batchduration call Dstreamgraph Generatejobs method, called Foreachdstream Generatejob method, It calls the Getorcompute method of the parent Dstream first to get the RDD, then in the calculation, from back to forward, the first dstream is Receiverinputdstream, Its Comput method obtains the metadata information of the corresponding time period from the Receivertracker, then generates the Blockrdd object and puts it into the Generatedrdds

Lecturing

From the last few lessons we know:

A. Spark streaming applications are ready to receive data at the beginning of processing data

B. When the application code for Spark streaming defines Dstream, one or more inputdstream are defined, and each inputdstream corresponds to one receiver

Combine the specific classes and methods of the source code to draw the receiver start full life cycle main flowchart:

(the original information from http://blog.csdn.net/andyshar/article/details/51476113, thanks to the author!) )

We start with the content of this talk, for everyone to analyze:

So what is the way receiver is supposed to start?

Receiver start-Up design problems analysis:
　
A. Spark streaming continuously receives data from external data sources via receiver and reports the data to the driver side, so that each batch durations can generate different jobs based on the reported data

B. It is possible to start multiple receiver in the same executor, in which case the load is uneven

C. Due to the failure of the executor operation itself, the task may fail to start and the entire job start fails, that is, receiver failed to start

D. Receiver is part of the spark streaming application startup phase and how it is designed to achieve receiver will always be started or started

E. Receivers and Inputdstreams is how one by one corresponds, by default, there is usually only one receiver?

Come on, go into the source together to see a really!!

Receiver start source thorough analysis:

How do I start receiver?

A. From the spark core point of view, receiver's start spark core does not know that receiver is started by job, runs on executor, and is run by task

B. In general, there is only one receiver, but you can create different data sources for Inputdstream

C. When the receiver is activated, a receiver is actually a partition and is initiated by a job with the transformations operation of the RDD and the action of the action, as the timer triggers, There is a constant generation of data reception, the received data generated in each time period is actually a partition

In this way, we are back to the problem of the initial receiver start-up scenario:

A. If there are multiple inputdstream, it is necessary to start multiple receiver, each receiver is equivalent to the Shard partition, then I start receiver when the ideal is to start receiver on different machines, but spark The core point of view is the application, not feel the specificity of receiver, so it will be in accordance with the normal job startup process, it is very likely to start multiple receiver on a executor, which can lead to load imbalance

B. It is possible to start receiver failure, as long as the cluster exists, receiver should not start failure

C. From the running process, a reveiver is a partition word, initiated by a task, if the task starts to fail, the receiver that starts with task will also fail

From this, it can be concluded that for receiver failure, the consequences are very serious, then spark streaming how to prevent these things?

Spark Streaming source Analysis, the following information is specified in spark streaming:

A. Spark uses a job to start a receiver. Maximum load Balancing

B. The Spark streaming has specified that each receiver runs on those executor and specifies where to run before receiver runs

C. If receiver fails to start, it is not a job failure at this time, the receiver will be restarted internally

Enter the StreamingContext source code, open the journey of decryption!

When the Start method of the StreamingContext is called, the Jobscheduler start method is called

Scheduler.start ()://Start child thread, on the one hand for local initialization work, on the other hand, do not block the main thread

In the Jobscheduler start method, the Receivertracker start method is called, and receiver initiates the

The Start method of the Receivertracker initiates the RPC message communication body, why? Because Receivertracker will monitor the entire cluster receiver,receiver turn around and report to receivertrackerendpoint about their status, the data they receive, including their lifecycle.

The specific receivers instances are obtained based on Receiverinputdstream (at the driver end) and then distributed to the worker node. A receiverinputdstream only corresponds to one receiver.

where Rundummysparkjob () to ensure that all nodes are alive and that all receivers are not concentrated on a single node

Go back and see Receivertracker.launchreceivers () in the Getreceiver ()

The Receiverinputdstream getreceiver () method returns the receiver object. The method is actually implemented in Receiverinputdstream subclasses.

Accordingly, this getreceiver () method must be implemented in Receiverinputdstream subclasses. The subclass of Receiverinputdstream must also define its own corresponding receiver subclass, because this receiver subclass is used in the Getreceiver () method to create an object of this receiver subclass.

Therefore, we need to look at the following Receiverinputdstream inheritance relationships

Based on the inheritance relationship, here's a look at the Getreceiver method in the subclass Socketinputdstream of Receiverinputdstream

The corresponding receiver sub-class Socketreceiver is also defined in the Socketinputdstream. The OnStart method must also be defined in the Socketreceiver class

The OnStart method starts a background thread and calls the Receive method

Start socket start receiving data

Go back to Receivertracker.launchreceivers () and look at the last Code endpoint.send (Startallreceivers (receivers)). This code sends a startallreceivers message to the Receivertrackerendpoint object, The Receivertrackerendpoint object is processed in receivertrackerendpoint.receive after it is received.

As you can see from the comments, spark streaming specifies which executors the receiver runs on, rather than on the task in Spark core to specify

Spark uses Submitjob to start receiver, and there will be a lot of receiver when the application executes, is this the time to start a receiver, or to start all receiver through this job?

The first parameter of the Startreceiver method in the Receive method of the Receivertracker is receiver, which shows that the for loop constantly pulls out receiver and then calls Startreceiver. This leads to the conclusion that a job only initiates a receiver.
　　
If receiver fails to start, this will not be considered a job failure and will resend the message to Receivertrackerendpoint to restart receiver, which will ensure that the receivers will be started. This will not be the case if a task initiates receiver if the failure is affected by the number of retries.

Receivertracker.startreceiver:

When receiver fails to start, it triggers receivertrackendpoint to restart a spark job to start receiver

The spark Job does not need to be restarted when receiver is closed

Look back at the code Supervisor.start () in Receivertracker.startreceiver. There is no Start method in subclass Receiversupervisorimpl, so the Start method of the parent class receiversupervisor is called.

Its specific implementation is the OnStart method of Receiversupervisorimpl in subclasses:

One of the _.start () is Blockgenerator.start:

Go back and look at the Startreceiver in Receiversupervisor.start ()

Still taking the subclass Socketreceiver of receiver as an example to illustrate the OnStart method

Socketreceiver.onstart:

This onstart method opens a thread that starts the socket to receive data. This running receive () is defined in Receiverinputdstream subclass Socketinputdstream

Note:
1. DT Big Data Dream Factory public number Dt_spark
2, Spark God-level experts: Liaoliang
3, Sina Weibo: Http://www.weibo.com/ilovepains

Spark Set-PLATE: 009~spark Streaming source interpretation of receiver in the driver of the subtle realization of the full life cycle of thorough research and thinking