Kafka Batch Collection Agent Configuration

You open the Kafka collection agent configuration dialog from a workflow configuration. To open the Kafka collection agent configuration, click Build → New Configuration. Select Workflow from the Configurations dialog. When prompted to Select workflow type, select Batch. Click Add agent and select Kafka from the Collection tab of the Agent Selection dialog.

Kafka batch collection agent configuration

Setting	Description
Profile	The name of the profile as defined in the Kafka Profile dialog (select Kafka Profile after clicking the New Configuration button in the Desktop).
All	If enabled, messages will be collected from all of the partitions.
Range	If enabled, messages will be collected from the range that you specify.
Specific	If enabled, messages will be collected from the specified partition(s). This is a comma separated list.
Batch Size	Use this field to set the number of messages you want to have in each batch. If this field is set to 0, all messages will be collected in one batch. Note! For performance reasons it is important to set a reasonable batch size. If the batch size is set too low, meaning that many batches are run per second, this will affect the performance.
Default Offset settings	The Kafka batch collection agent will collect the messages in sequence, and when the batch is finished the last number in the sequence will be the offset for the messages in the next batch. If the next message in sequence does not exist in the next batch for some reason, the Default Offset settings can be used to determine where the Kafka batch collection should start.
Start at beginning	If you select this option, the agent will start collecting from the first message in the queue.
Start at end	If you select this option, the agent will start collecting from the last message in the queue. This means that the workflow will not collect anything, but the offset will be updated to the last messages for the next run.