Spark Version Custom 7th day: Jobscheduler Insider realization and deep thinking

Source: Internet
Author: User

Contents of this issue:

1 Jobscheduler Insider Realization

2 Deep thinking

All data that cannot be streamed in real time is invalid data. In the stream processing era, Sparkstreaming has a strong appeal, and development prospects, coupled with Spark's ecosystem, streaming can easily call other powerful frameworks such as Sql,mllib, it will eminence.

The spark streaming runtime is not so much a streaming framework on spark core as one of the most complex applications on spark core. If you can master the complex application of spark streaming, then other complex applications are a cinch. It is also a general trend to choose spark streaming as a starting point for custom versions.

In the job generation method, Jobgenerator will be dynamically generated jobset every batchinterval time submitted to Jobscheduler

Private def generatejobs (time:time) {  //Set The sparkenv in this thread, so that job generation code can access the Environment  //Example:blockrdds is created in this thread, and it needs to access Blockmanager  //Update:this is probably redundant after threadlocal stuff in sparkenv have been removed.  Sparkenv.set (ssc.env)  Try {    jobScheduler.receiverTracker.allocateBlocksToBatch (time)//Allocate received Blocks to Batch    graph.generatejobs (time)//generate jobs using allocated block  } match {case    Success (jobs) = >      val streamidtoinputinfos = jobScheduler.inputInfoTracker.getInfo (time)      Jobscheduler.submitjobset ( Jobset (Time, Jobs, Streamidtoinputinfos))//Submit Jobset Case    Failure (e) =      Jobscheduler.reporterror (" Error Generating jobs for Time "+ Time, E)  }  eventloop.post (Docheckpoint (time, clearcheckpointdatalater = Fals e))}

In the Submitjobset method, a jobhandler is generated for each job and is given to Jobexecutor to run.

def submitjobset (jobset:jobset) {   if (jobSet.jobs.isEmpty) {     loginfo ("No jobs added for time" + jobset.time) 
   

  

Private Val jobexecutor =  Threadutils.newdaemonfixedthreadpool (numconcurrentjobs, "Streaming-job-executor")

In the generated thread pool, the generated jobhandler are used to handle the event. In this case Jobhandler will call Job.run (), it will trigger the real execution of Job.func! The job will start running here.

Note:

Data from: Dt_ Big Data Dream Factory (spark release version customization)

For more private content, please follow the public number: Dt_spark

If you are interested in big data spark, you can listen to it free of charge by Liaoliang teacher every night at 20:00 Spark Permanent free public class, address yy room Number: 68917580

Spark Version Custom 7th day: Jobscheduler Insider realization and deep thinking

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.