Flink Kafka producer with transaction support

Background

In Flink 1.5 above, it provides a new Kafka producer implementation:flinkkafkaproducer011, aligning with Kafka 0.11 above that supports transaction. Kafka transaction allows multiple Kafka messages sent by producer to deliver on an atomic the-and either all success or All fail. The messages can belong to different partitions. Before Flink 1.5, it provides exact once semantics only for its internal state via check point. However, if you write the stream state to external storage such as Kafka or database, and there is no guarantee. In Flink 1.5, an important new class twophasecommitsinkfunction are introduced to support extactly once semantic f or Externa storage. FlinkKafkaProducer011 leverage this feature, so before we are at FlinkKafkaProducer011, we'll briefly go through what T Wophasecommitsinkfunction does.

Twophasecommitsinkfunction

As its name suggests, Twophasecommitsinkfunction provides a template to implement the phase commit protocol so that Writin g to external storage and Flink checkpoint reaches consensus on success/failure. IT provides the following abstract methods:

• BeginTransaction
• Precommit
• Commit
• Abort
• Invoke (TXN transaction, in value, context context)

The sequence diagram below shows how these methods is invoked (highlighted in red)

In a typical and phase commit protocol for database, the precommit operation usually involves writing DB operations to a W AL file (write ahead log). Once The DB operation is persisted, even if server crashed afterwards, it can all recover db operations from log and AP Ply to database, so commits can always success.

Internally, Twophasecommitsinkfunction keeps pendingtransactions : An Orderedmap (Linkedhashmap) that records Current pending transactions This not yet committed. Key is the checkpoint ID of that transaction occurs, value was the transaction holder object that contains transaction start T IME and transaction information. Normally the map contains only one transaction state entry corresponding to the current checkpoint. Under rare situation, it can contain transaction state entry from previous checkpoint if previous checkpoint failed or ACK Nowledgement is delayed.

On snapshot State (snapshotstate):
Current transaction holder was put into pendingtransactions. Precommit is called to perform the prepare step in the phase commit. A new transaction is started and internal are updated to reflect the latest current transaction holder and Pendingtra Nsactions.

On Checkpoint notification (Notifycheckpointcomplete):
All pending transactions in Pendingtransactions that has ID less than or equals the notified checkpoint ID would be commit Ted. Committed transaction is removed from Pendingtransactions.

Checkpoint Recovery (initializestate)
Recoverandcommit method would be called on each of the pendingtransactions. While for the current transaction in state (Precommit method not yet called), Recoverandabort method is called

To summarize:
Transaction scope aligns with the checkpoint frequency, indicating-all records between, commits would belong to a s Ingle transaction.
Prepare (Precommit) step must succeed for snapshot to complete.

Kafka Transaction API

Kafkaproducer.inittransactions ()
Kafkaproducer.begintransaction ()
Kafkaproducer.sendoffsetstotransaction (offsets, consumergroupid)
Kafkaproducer.committransaction ()
Kafkaproducer.aborttransaction ()

Besides a special property "transactional.id" needs to being assigned to Producerconfig. This raises a important implication that there can is only one active transaction per producer at any time.

Inittransactions Method:ensures Any transactions initiated by previous instances of the producer with the same transactio Nal.id is completed. If the previous instance had failed with a transaction in progress, it'll be aborted. If the last transaction had begun completion, but not yet finished, this method awaits its completion

Also:inittransactions method assigns an internal producer ID and epoch number. Each producer restart would generate a new epoch number, thus allow Kafka transaction manager to detect and reject transact Ion request that was sent before producer restart.

Sendoffsetstotransaction method:used when read from one Kafka topic and send data to a other topic (Eg, Kafka streaming) . So, consumer offset shifts only if producer successfully send data (exactly once)

Combine both

FlinkKafkaProducer011 supports 3 modes:exactly once, at least once or none. Under exactly once mode, a new Kafkaproducer would be created on each call to Begintransa Ction whereas under the other of the modes, existing kafkaproducer from current state would be reused. Another important difference is this Kafka transaction is only enabled for exactly once mode. As discussed previously, BeginTransaction method can be called from Initializestate method or Snapshotstate method. A new kafkaproducer with a different transaction ID ensures that producer records sent between the checkpoints belong to D Ifferent transactions, so-one can be committed or rolled-back separately.

The difference between at least once or none mode is the precommit behavior. For exactly once and at least once mode, kafkaproducer are flushed in precommit to make sure that records in producer Buffe R is sent to broker before checkpoint snapshot completes. Otherwise, if Flink crashes after the current checkpoint, unsent records in producer buffer is lost.

Transaction State object:kafkatransactionstate contains a transient flink Kafka producer. Transaction ID, producer ID and epoch. The Kafka Producer Reference Act as a shortcut to retrieve Kafka producer from current Transactionholder in various Lifecy CLE stage. On the recovery from checkpoint, Kafka producer in transaction state would be null and need to be re-created. To commit, transaction ID, producer ID and epoch for the new Kafka producer need to match the previous Kafka producer Befo Re crash. Otherwise, Broker cannot associate commit request with the producer for previous transaction and reject it. See flinkkafkaproducer.resumetransaction method in how to transaction state, producer ID and epoch is set. To abort, only transaction ID matters, as there can is only one pending transaction per transaction ID.

Another interesting aspect worth exploration is a unique transaction ID is generated. As Flink Kafka producer runs in parallel, generating a distributed, yet globally unique transaction ID is a challenge.
The transaction ID can is recycled after commit, so we don't need an unbounded sequence but a range of unique values.

The ID range generated'll is from startIndex + subtaskIndex * poolSize To startIndex + (subtaskIndex + 1) * poolSize - 1
Start index is stored in Nexttransactionalidhint on Flink list state object. Each time a new transaction ID range is requested, it'll be incremented by NumberOfParallelSubtasks * kafkaProducersPoolSize;
But why isn't just use Java UUID? I think there is and reasons:
Java UUID is purely random, where a monotonically increasing transaction ID sequence are more desirable for tracking and MO nitoring purpose.
If Flink crashes before the first checkpoint complete, there are no information about transaction ID in state. In this case, system have to guess possible transaction IDs within a range and try to abort each of the them. With purely random uuid, even guess was not possible.

Other Thoughts

Currently, only transaction ID was pooled for reuse. For exactly once mode, a new Kafka producer are created on each checkpoint snapshot. Depending on the checkpoint interval, this might is an overhead. Why isn't pool Kafka producer directly instead?

Flink Kafka producer with transaction support