Flink status and fault tolerance (state and Fault tolerance)

Flink state and fault tolerance

Stateful functions and operations store data by processing a single (element/event), allowing any type of state to build more complex operations.

For example:

When the application searches for a particular event pattern, state stores some of the column events so far. When events are aggregated per minute/hour/day, state holds all data waiting to be aggregated. When the machine learning model is trained on a series of data streams, State holds the model parameters for the current version. State allows efficient access to past data when historical data needs to be managed

Flink can use checkpoints to perform fault-tolerant management of statue and allow a streaming application to execute savepoint.

Knowing that the state of the application can be re-tuned means that Flink is responsible for reassigning the status of the parallel instances.

Flink's queryable state allows you to receive state data externally while the Flink is running.

When working with state, it is recommended to read Flink's state backends. Flink provides a different state backends to specify how the state data is stored and stored. The state can exist in the Java heap. Depending on your state Backends,flink can also manage the status of your application, meaning that the Flink can run the application to save very large states through memory management (which can overflow to disk if necessary). The state backends can be configured without changing the logic of the application.

There will be a corresponding feature article explaining the following content

Working with states: shows how to use state in the Flink application and explains the different types of status checkpointing: A checkpoint that describes how to start and configure fault tolerance queryable State: Explains how to access a state other than Flink at run time custom serialization for managed state: Discusses the custom serialization logic of state and its upgrade