You open the Kafka collection agent configuration dialog from a workflow configuration. Click Build → New Configuration. Select Workflow from the Configurations dialog. When prompted to Select workflow type, select Batch. Click Add agent and select Kafka from the Collection tab in the Agent Selection dialog. Double-click on the agent in the workflow template.
Setting | Description | ||||||
|---|---|---|---|---|---|---|---|
Consumer Group | The name of the consumer group which is defined by the | ||||||
Batch Size | The number of messages to collect per batch. Note that a workflow will remain in running state even when all data has been consumed and you have to trigger a stop either using mzcli, or the operations REST interface or in the workflow configuration by using a MIM like Estimated Lag or Workflow Throughput, for example. The Batch Size must be greater than 0. Note! For performance reasons, it is important to set a reasonable batch size. If the batch size is set too low, this will affect the performance negatively. | ||||||
Assignment | Messages can be collected from one or several topics. You define how the topics should be identified by selecting either; Topic Pattern Topic Names Topic Partitions Below are few examples of valid partition declarations: Example of collection from partition 0:
Example of collection from the three partitions 0, 8 and 12:
Example of collection from the six partitions 0, 3, 4, 5, 6, and 7:
Note! If you select Topic Partitions, automatic rebalancing will not take place, and you will have to handle potential rebalancing manually if needed. |
Note!
You must configure a Kafka Profile in the Execution tab in Workflow Properties for the workflow configuration to be valid.