New Kafka Agents

The Kafka Agents differ significantly from other agents in their design, as the workflow transactions are stored in the Kafka cluster and not in the platform database as for other agents. Therefore, you can only configure one Kafka cluster per workflow for batch workflows. The transactions are synchronized per workflow and stored in the associated cluster when the workflow contains a Kafka agent.

If you want to collect messages from one Kafka cluster and forward them to another cluster using batch workflows, you must configure two workflows. Another difference for the batch agents is that you select the Kafka profile in the Execution Tab of the Workflow Properties dialog rather than in the agent configuration as you do for the new real-time, and old legacy agents.

Note!
This is a new set of Kafka agents that will replace the previous agents in a future major version. In Desktop, the previous agents and profile are marked as deprecated and have the following names:

Kafka Profile Deprecated
Batch collection agent: Kafka Deprecated
Realtime collection agent: Kafka Deprecated
Realtime forwarding agent: Kafka Deprecated

The main differences between the versions are that the new version:

Includes a batch forwarding agent.
Stores transactions in the Kafka cluster (transactions are only tracked for batch workflows).
Stores offset in the Kafka cluster only.
One consumer can collect from several topics, and one producer can forward to several topics.
Includes automatic rebalancing for collection of messages. Several workflows can collect messages in parallel from the same topics, provided that there are several partitions. When you start a new workflow, the running workflows will automatically be rebalanced, see Automatic Scale Out and Rebalancing.
Batch collection workflows will remain in running state, even when all data has been collected.
Uses only two UDR types:
- kafkaRecord for both collection and forwarding agents.
- kafkaOffset for resetting of offset.

As the UDR types differ between the deprecated and new agents, you have to rewrite workflows with Legacy Kafka agents after having upgraded to this version.

Mandatory Topics

When using Kafka agents in batch workflows, transactions are stored in two Kafka topics: wf_txn and txn_status.

When the workflow starts, the system checks whether these topics exist. If they do not, it attempts to create them. However, if the SASL user lacks permission to create topics—such as when restricted by Kafka ACLs—they must be created beforehand.

Warning!
Do not modify the wf_txn or txn_status topics after they have been created. This includes adding partitions, changing properties, or altering definitions. Doing so may result in unexpected behavior.

Topic requirements:

The topics must be named wf_txn and txn_status.
They must have the property cleanup.policy=compact.
Each topic should have only one partition.

Example - creating Kafka topics

kafka-topics.sh --create \
                --bootstrap-server <HOST>:<PORT> \
                --config cleanup.policy=compact \
                --partitions 1 \
                --topic <TOPIC_NAME>