Versions Compared

Old Version 4

changes.mady.by.user Felicity Borrelli

Saved on

compared with

New Version 5

changes.mady.by.user Felicity Borrelli

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The Kafka batch collection agent consumes messages from the topic and partitions stated in the  Kafka the Kafka collection agentsagent's configuration, which contains the following settings:

Setting

Description

Profile

Browse and select the profile the agent should use, as defined in the Legacy Kafka Profile.

All

Select this option if you want messages to be be collected from all of the partitions.

Range

Select this option if you want messages to be collected from the range that you specify.

Specific

Select this option if you want messages to be collected from the specified partition(s). This is a comma-separated list.

Batch Size

Use this field to set the number of messages you want to have in each batch. If this field is set to 0, all messages will be collected in one batch. 

Note

Note!

For performance reasons, it is important to set a reasonable batch size. If the batch size is set too low, meaning that many batches are run per second, this will affect the performance. 

Default Offset settings
The Kafka batch collection agent will collect the messages in sequence, and when the batch is finished the last number in the sequence will be the offset for the messages in the next batch. If the next message in sequence does not exist in the next batch for some reason, the Default Offset settings can be used to determine where the Kafka batch collection should start. 

Start at Beginning

Select this option if you want the agent to start collecting from the first message in the queue. 

Start at End

Select this option if you want the agent to start collecting from the last message in the queue. This means that the workflow will not collect anything, but the offset will be updated to the last messages for the next run. 

...