Kafka Real-time Collection Agent Configuration
- Kevin Wu
You open the Kafka collection agent configuration dialog from a workflow configuration. To open the Kafka collection agent configuration, click  Build → New Configuration. Select Workflow from the Configurations dialog. When prompted to Select workflow type, select Realtime. Click Add agent and select Kafka from the Collection tab of the Agent Selection dialog.
Kafka collection agent configuration
Setting | Description |
|---|---|
Profile | The name of the profile as defined in the Kafka Profile dialog (select Kafka Profile after clicking the New Configuration button in the Desktop). |
All | If enabled, messages will be collected from all of the partitions. |
Range | If enabled, messages will be collected from the range that you specify. |
Specific | If enabled, messages will be collected from the specified partition(s). This is a comma-separated list. |
Start at beginning | You must determine from which offset you want to start collecting. If enabled, messages are collected from the first offset. If you select this option, there is a risk that messages will be processed multiple times after a restart. |
Start at end | You must determine from which offset you want to start collecting. If enabled, messages are selected from the last offset from when the workflow was started. If you select this option, there is a risk that data can be lost after a restart. |
Start at requested | You must determine from which offset you want to start collecting. If enabled, you must add an incoming route for the UDRs, and when a UDR arrives on that route - collection starts from the given offset. You set the offset using the |
Managed by Kafka (auto commit) | If enabled, the offset is saved by the Kafka server. In this case, the Kafka server knows where to start. |