Kafka Real-time Collection Agent Configuration

You open the Kafka collection agent configuration dialog from a workflow configuration. To open the Kafka collection agent configuration, click  Build → New Configuration. Select Workflow from the Configurations dialog. When prompted to Select workflow type, select Realtime. Click Add agent and select Kafka from the Collection tab of the Agent Selection dialog.

Kafka collection agent configuration

Setting

Description

Setting

Description

Profile

The name of the profile as defined in the Kafka Profile dialog (select Kafka Profile after clicking the New Configuration button in the Desktop).

All

If enabled, messages will be collected from all of the partitions.

Range

If enabled, messages will be collected from the range that you specify.

Specific

If enabled, messages will be collected from the specified partition(s). This is a comma-separated list.

Start at beginning

You must determine from which offset you want to start collecting. If enabled, messages are collected from the first offset. If you select this option, there is a risk that messages will be processed multiple times after a restart.

Start at end

You must determine from which offset you want to start collecting. If enabled, messages are selected from the last offset from when the workflow was started. If you select this option, there is a risk that data can be lost after a restart.

Start at requested

You must determine from which offset you want to start collecting. If enabled, you must add an incoming route for the UDRs, and when a UDR arrives on that route - collection starts from the given offset. You set the offset using the KafkaOffsetUDR in an Analysis agent, see KafkaOffsetUDR. Using this offset option reduces the risk of data loss, and prevents messages from being processed multiple times after a restart. See the example in the KafkaOffsetUDR Example in KafkaOffsetUDR.

Managed by Kafka (auto commit)

If enabled, the offset is saved by the Kafka server. In this case, the Kafka server knows where to start.