New Kafka Agents

The Kafka Agents differ significantly from other agents in their design, as the workflow transactions are stored in the Kafka cluster and not in the platform database as for other agents. Therefore, you can only configure one Kafka cluster per workflow for batch workflows. The transactions are synchronized per workflow and stored in the associated cluster when the workflow contains a Kafka agent.

If you want to collect messages from one Kafka cluster and forward them to another cluster using batch workflows, you must configure two workflows. Another difference is that you select the Kafka profile in the Workflow Properties dialog and not in the agent configuration as you do for other agents with dedicated profiles.

This is a new set of Kafka agents that will replace the previous agents in a future major version. In Desktop Online, the previous agents and profile are marked as deprecated and have the following names:

Kafka Profile Deprecated
Batch collection agent: Kafka Deprecated
Realtime collection agent: Kafka Deprecated
Realtime forwarding agent: Kafka Deprecated

The main differences between the versions are that the new version:

Includes a batch forwarding agent.
Stores transactions in the Kafka cluster (transactions are only tracked for batch workflows).
Stores offset in the Kafka cluster only.
One consumer can collect from several topics, and one producer can forward to several topics.
Includes automatic rebalancing for collection of messages. Several workflows can collect messages in parallel from the same topics, provided that there are several partitions. When you start a new workflow, the running workflows will automatically be rebalanced, see Automatic Scale Out and Rebalancing.
Batch collection workflows will remain in running state, even when all data has been collected.
Uses only two UDR types:
- kafkaRecord for both collection and forwarding agents.
- kafkaOffset for resetting of offset.

As the UDR types differ between the deprecated and new agents, you have to rewrite workflows with Legacy Kafka agents after having upgraded to this version.