"Flink Series 2" Time Window _flink

Leads

For streaming systems, the incoming message is infinite, so for operations such as aggregation or connection, the streaming system needs to segment incoming messages and then aggregate or connect based on each piece of data. The segmentation of the message is called a window, the streaming system supports many types of windows, the most common is the time window, based on the time interval for the message segmentation processing. This section focuses on the various time Windows supported by the Flink streaming system.

For most current streaming systems, the time window is usually split according to the local clock of the task's node, which is easier to implement and does not block message processing. However, some application requirements may not be met, for example:

1. The message itself has a timestamp, and the user wants to segment it according to the time characteristics of the message itself.

2. Because the clocks of different nodes may be different and the delay is different when the message flows through each node, the message processed by a node in the same time window may be split into a different time window when it flows to the next node, resulting in a result that does not meet the expected results.

Three kinds of time windows

Flink supports three types of time windows, which apply to the different types of time windows required by the user:

1. Operator time. A time window that is split according to the local clock of the node on which the task resides.

2. Event time. The message is time-stamped and processed according to the timestamp of the message to ensure that all messages in the same time window must be handled correctly. Because a message can flow into a task in random order, the task needs to cache the state of the current time window message processing until all messages belonging to that time window are processed before they can release their state. If the chaotic message latency is high, it can affect the throughput and latency of the distributed system.

3. Ingress time. Sometimes the message itself does not have timestamp information, but the user still wants to divide the time window by the message instead of the node clock (for example, to avoid the second problem mentioned above). At this point, when the message source flows into the Flink stream processing system, the timestamp of the increment is automatically generated, and the process is followed by the same as event time. Ingress time can be regarded as a special case of event times, because its timestamp at the source must be ordered, so in a streaming system, the message delay is not very high in the stream processing, relative to the event, Consequently, the impact on throughput and latency of flink distributed systems will be even smaller.

Implementation of the Event time window

Flink borrowed from Google's Millwheel project and watermark to support the event time window. When an operator processes data by using a time window based on event times, it must be able to start processing data only after it has been determined that all messages belonging to that time window have been streamed into the operator. However, because the message may be disorderly, the operator cannot directly confirm when all messages belonging to that time window will flow into this operator.

Watermark contains a timestamp, Flink uses watermark to mark all messages that are less than the timestamp have been streamed in.

After the Flink data source confirms that all messages less than a timestamp have been exported to the Flink stream processing system, a watermark containing the timestamp is generated, inserted into the message flow to the Flink stream processing system, and the Flink operator caches all incoming messages according to the time window. When the operator is processed to watermark, it processes all data that is less than the time window of the watermark timestamp and sends it to the next operator node, and then watermark to the next operator node.

In order to be able to handle all messages belonging to a certain time window, the operator must wait for the watermark of the time window to begin processing the message for that time window, and Flink need to occupy more memory than the operator time window. and will directly affect the delay time of message processing. One possible optimization measure for this is that for an operator of an aggregation class, it may be possible to aggregate some of the messages in advance, and when there is a new message flowing into that window, the calculation is based on the previous partial aggregation result, so that only the intermediate calculation results can be cached without caching all the messages for the time window.

Sort based on time stamps

In a streaming system, sorting messages is generally considered infeasible because the incoming message is infinite. But in the Flink flow processing system, based on the Watermark,flink, the global ordering based on time stamp is realized.

Flink the realization of sorting based on timestamp is as follows: the sort operator caches all incoming messages and, when it receives watermark, sorts messages that are less than that watermark and sends to the next node. Releases all timestamps that are less than the watermark in this sort operator, continues to cache incoming messages, waits for the next watermark to trigger the next order.

Because watermark guarantees that it will not appear after the time stamp is smaller than the message, so you can guarantee the correctness of the order. It should be noted that if the sort operator has more than one node, only to ensure that the outgoing message of each node is ordered, the message between the nodes is not guaranteed orderly, to achieve global order, there can only be a sort operator node.

By supporting message processing based on event time, Flink expands the application scope of its streaming processing system, so that more flow processing tasks can be performed through Flink.

Reference:

Http://bigdata.51cto.com/art/201702/531036.htm