Profile

Browse and select the profile the agent should use, as defined in Kafka (4.1).

All

Select this option if you want messages to be collected from all of the partitions.

Range

Select this option if you want messages to be collected from the range that you specify.

Specific

Select this option if you want messages to be collected from the specified partition(s). This is a comma separated list.

Offset
You must determine from which offset you want to start collecting.

Start at Beginning

 Select this option to collect messages from the first offset. If you select this option, there is a risk that messages will be processed multiple times after a restart .

Start at End

Select this option to select messages from the last offset from when the workflow was started. If you select this option, there is a risk that data can be lost after a restart .

Start at Requested

Select this option to start collection from the given offset on the incoming route specified in the UDRs. You set the offset using the KafkaOffsetUDR in an Analysis agent, see KafkaOffsetUDR(4.1). Using this offset option reduces the risk of data loss, and prevents messages from being processed multiple times after a restart. See the example in the KafkaOffsetUDR Example in KafkaOffsetUDR(4.1).

Managed by kafka (auto commit)

This function should be used in the case you have a kafka profile using the Group id, to enable load sharing between several Kafka collectors. Use this in combination with enable.auto.commit=true in the Kafka profile, in the Advanced, Consumer tab. If you select this option, the consumed messages will committed intermittently which may result in lost or duplicate data. To avoid this, select the Start at Requested option instead.