Kafka Real-Time Collector(3.1)

Owned by Jenni Wallin
May 10, 2023
2 min read

The Kafka real-time collection agent consumes messages from the topic and partitions stated in the Kafka collection agents' configuration, which contains the following settings:

SettingDescription

Profile

Browse and select the profile the agent should use, as defined in the Kafka (3.0).

All

Select this option if you want messages to be collected from all of the partitions.

RangeSelect this option if you want messages to be collected from the range that you specify.
SpecificSelect this option if you want messages to be collected from the specified partition(s). This is a comma separated list.
Offset
You must determine from which offset you want to start collecting.
Start at Beginning

 Select this option to collect messages from the first offset. If you select this option, there is a risk that messages will be processed multiple times after a restart .

Start at End

Select this option to select messages from the last offset from when the workflow was started. If you select this option, there is a risk that data can be lost after a restart .

Start at Requested

Select this option to start collection from the given offset on the incoming route specified in the UDRs. You set the offset using the KafkaOffsetUDR in an Analysis agent, see KafkaOffsetUDR(3.1). Using this offset option reduces the risk of data loss, and prevents messages from being processed multiple times after a restart. See the example in the KafkaOffsetUDR Example in KafkaOffsetUDR(3.1).

Managed by kafka (auto commit)This function should be used in the case you have a kafka profile using the Group id, to enable load sharing between several Kafka collectors. Use this in combination with enable.auto.commit=true in the kafka profile, in the advanced, consumer tab.

Input/Output Data

Input Data

Messages from the Kafka Log.

KafkaOffsetUDR(3.1)

Output Data

KafkaUDR(3.1)