What is the stage & job & task of Spark, and the principle of Division __spark

These concepts are easily confusing and need to be written over the article to comb

What is Spark's stage & job & task, and how it is divided

Stage is a very important concept in spark,

An important basis for dividing stage in a job is whether a shuflle occurs, that is, whether the data is reorganized (reorganizing the data).

A lot of tasks are executed within a stage, and after all tasks in the same stage, the task in the next stage can be relied upon DAG.

A job has a number of tasks, each set of tasks can be a stage
Task is another important concept in spark,

Tasks are closely linked to concepts such as the partition block, which are the logical units that execute the job, and are executed in the CPU core of each executor in the task
a Job is a larger logical concept than task and stage,

Job can be thought of as an action in driver or a program submitted by Spark-submit, and there are many action in our program, all of which correspond to a lot of jobs.