Kafka Real-Time Collector(3.1)
The Kafka real-time collection agent consumes messages from the topic and partitions stated in the Kafka collection agents' configuration, which contains the following settings:
Setting | Description |
---|---|
Profile | Browse and select the profile the agent should use, as defined in the Kafka (3.0). |
All | Select this option if you want messages to be collected from all of the partitions. |
Range | Select this option if you want messages to be collected from the range that you specify. |
Specific | Select this option if you want messages to be collected from the specified partition(s). This is a comma separated list. |
Offset You must determine from which offset you want to start collecting. | |
Start at Beginning | Select this option to collect messages from the first offset. If you select this option, there is a risk that messages will be processed multiple times after a restart . |
Start at End | Select this option to select messages from the last offset from when the workflow was started. If you select this option, there is a risk that data can be lost after a restart . |
Start at Requested | Select this option to start collection from the given offset on the incoming route specified in the UDRs. You set the offset using the |
Managed by kafka (auto commit) | This function should be used in the case you have a kafka profile using the Group id, to enable load sharing between several Kafka collectors. Use this in combination with enable.auto.commit=true in the kafka profile, in the advanced, consumer tab. |
Input/Output Data
Input Data
Messages from the Kafka Log.