Spark Structured streaming Framework (5) Process Management

Source: Internet
Author: User
Tags unique id

Structured streaming provides some APIs to manage streaming objects. These APIs allow users to manually manage the streaming that have been started, ensuring that the streaming in the system is executed in an orderly manner.

1. Streamingquery

A Streamingquery object is returned after the start streaming of the Datastreamwriter method is called. So the user can manage the streaming through this object.

As shown below:

Val query = Df.writeStream.format ("console"). Start ()//Get the Query object

Query.id//Get the unique identifier of the running query that persists across restarts from checkpoint data

Query.runid//Get the unique ID of this run of the query, which'll be generated at every Start/restart

Query.name//Get the name of the auto-generated or user-specified name

Query.explain ()//print detailed explanations of the query

Query.stop ()//Stop the query

Query.awaittermination ()//block until query is terminated, with stop () or with error

Query.exception//The exception if the query has a been terminated with error

Query.recentprogress//An array of the most recent progress updates for this query

Query.lastprogress//The most recent progress update of this streaming query

2. Streamingquerymanager

Structured streaming provides another interface for managing streaming: Streamingquerymanager. The user can get through the streams method of the Sparksession object.

As shown below:

Val spark:sparksession = ...

Val Streammanager = Spark.streams ()

Streammanager.active//Get the list of currently active streaming queries

Streammanager.get (ID)//Get a Query object by its unique ID

Streammanager.awaitanytermination ()//block until any one of them terminates

3. References

[1].structured streaming Programming Guide.

[2]. Kafka IntegrationGuide.

Spark Structured streaming Framework (5) Process Management

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.