Get Spark Program execution progress and exceptions via DEVELOPERAPI

Effect Display:

Code:

 PackageOrg.apache.spark.zpc.listenerImportorg.apache.spark.LoggingImportorg.apache.spark.scheduler._Importscala.collection.mutable/*** Spark's DEVELOPERAPI provides execution monitoring for app, job, and task.  * Through this monitoring, can achieve: * 1. Task execution progress rough calculation.  * 2. Get exception information when performing an exception failure.  * 3. Get the app-initiated AppID so you can control the kill task. * 4. Handle processing of custom progress and exceptions (such as console printing, saving db, or JMS transmission to the terminal such as the Web * *@paramJobnum The number of jobs in application.  You can see the spark log by submitting the code. */Abstract classSparkapplistener (Jobnum:int)extendsSparklistener with Logging {//mapping of job and job information (including total task count, current completed task count, current job percentage)  PrivateVal Jobtojobinfo =Newmutable. Hashmap[int, (int, int, int)]//Stageid and job mappings, the user gets the job corresponding to the task  PrivateVal Stagetojob =Newmutable. Hashmap[int, Int]//number of job completed  Privatevar finishjobnum = 0Privatevar Hasexception:boolean =falseoverride Def Onapplicationstart (Applicationstart:sparklistenerapplicationstart): Unit=synchronized{val appId=Applicationstart.appid//record the ID of the app for subsequent processing://such as: Yarn Application-kill appId//handleappid (appId)  }  //Gets the task number of the job, initializes the job informationOverride Def onjobstart (jobstart:sparklistenerjobstart) =synchronized{val jobId=Jobstart.jobid val Tasks= JobStart.stageInfos.map (Stageinfo =stageinfo.numtasks). Sum Jobtojobinfo+ = (jobId (tasks, 0, 0)) JobStart.stageIds.map (Stageid= = Stagetojob (Stageid) =jobId)} //at the end of the task, roughly estimate the current app execution progress. //Estimation Method: The currently completed task quantity/Total task quantity. Total number of completed tasks by (total jobs * The number of tasks for the current job). )Override Def ontaskend (taskend:sparklistenertaskend) =synchronized{val Stageid=Taskend.stageid Val jobId=Stagetojob.get (Stageid). Get Val (Totaltasknum:int, Finishtasknum:int, Percent:int)=Jobtojobinfo.get (jobId). Get Val currentfinishtasknum= Finishtasknum + 1Val newpercent= Currentfinishtasknum */(Totaltasknum *jobnum) jobtojobinfo (jobId)=(Totaltasknum, Currentfinishtasknum, newpercent)if(Newpercent >percent) {      //Hanlde Application ProgressVal totalpercent =JobToJobInfo.values.map (_._3). Sumif(Totalpercent <= 100){//handleappprogress (totalpercent)      }    }  }  //The job ends, gets the status of the job end, and returns the type of exception to handle. //handle handles customizations, such as returning to the web side, showing the Exception log. Override Def onjobend (jobend:sparklistenerjobend) =synchronized{jobend.jobresult Match { Casejobsucceeded = Finishjobnum + = 1 CaseJobfailed (Exception)if!hasexception =hasexception=true        //Handle Application Failure//handleappfailure (Exception)       Case_ =    }  }  //when the app finishes, mark the program execution progress as 100%. //defect: Sparklistenerapplicationend does not provide exception access to the app. This way, when the program is faulted on the driver side,//Gets no specific reason for the error to return to the front end, custom prompt. For example (driver to the application of SQL parsing exception, has not started the job run)

/*** driver-side exceptions can be obtained through a try catch in the main program code ***/
Override Def onapplicationend (applicationend:sparklistenerapplicationend) =synchronized{val Totaljobnum=jobToJobInfo.keySet.size Val totalpercent=JobToJobInfo.values.map (_._3). Sum//Handle Precision Lose    if(!hasexception && totalpercent = = 99) {//handleappprogress (+)} val msg= "Execution Failed"if(Totaljobnum = = 0){handleappfailure (New Exception (msg))    }  }}

Get Spark Program execution progress and exceptions via DEVELOPERAPI