Spark streaming is a spark-based stream batch processing engine. Its basic principle is to process input data in batches at a certain interval. When the batch processing interval is shortened to seconds, it can be used to process real-time data streams.
This section describes the execution process of spark streaming jobs.
Figure 1 execution process of spark streaming jobs
Specific process:
- Start the driver after the client submits the job. The driver is the master of the park job.
- Each job contains multiple executors, and each executor runs tasks in a thread mode. Spark streaming contains at least one executor task.
- The receiver generates a block after receiving the data, reports the blockid to the driver, and backs up the block to another executor.
- Receivertracker maintains the blockid reported by the reciver.
- The driver regularly starts jobgenerator, generates logical RDD based on the relationship of dstream, creates a jobset, and submits it to jobscheduler.
- Jobschedset schedules jobset and delivers it to dagscheduler. dagschedages generates stages based on the Logical RDD. Each stage contains one or more tasks.
- Taskschedtor schedules tasks to executor and maintains the running status of tasks.
- After task, stages, and jobset are complete, a single batch is complete.
Spark streaming Execution Process
