Kafka Real-time Collection Agent Configuration

You open the Kafka collection agent configuration dialog from a workflow configuration. To open the Kafka collection agent configuration, click Build → New Configuration. Select Workflow from the Configurations dialog. When prompted to Select workflow type, select Realtime. Click Add agent and select Kafka from the Collection tab of the Agent Selection dialog.

Kafka collection agent configuration

Setting	Description
Profile	The name of the profile as defined in the Kafka Profile dialog (select Kafka Profile after clicking the New Configuration button in the Desktop).
All	If enabled, messages will be collected from all of the partitions.
Range	If enabled, messages will be collected from the range that you specify.
Specific	If enabled, messages will be collected from the specified partition(s). This is a comma-separated list.
Start at beginning	You must determine from which offset you want to start collecting. If enabled, messages are collected from the first offset. If you select this option, there is a risk that messages will be processed multiple times after a restart.
Start at end	You must determine from which offset you want to start collecting. If enabled, messages are selected from the last offset from when the workflow was started. If you select this option, there is a risk that data can be lost after a restart.
Start at requested	You must determine from which offset you want to start collecting. If enabled, you must add an incoming route for the UDRs, and when a UDR arrives on that route - collection starts from the given offset. You set the offset using the `KafkaOffsetUDR` in an Analysis agent, see KafkaOffsetUDR. Using this offset option reduces the risk of data loss, and prevents messages from being processed multiple times after a restart. See the example in the KafkaOffsetUDR Example in KafkaOffsetUDR.
Managed by Kafka (auto commit)	If enabled, the offset is saved by the Kafka server. In this case, the Kafka server knows where to start.