Example - Collect Messages from 2 Topics with 3 Configured Partitions You want to collect messages from 2 Kafka topics called “example1” and “example2” which have 3 configured partitions each. In the Kafka collection agent configuration, you only need to state the names of the topics and consumer group, and the agent will automatically manage the assignment of partitions within the consumer group. You will get the following behavior and debug in the executing workflows: Code Block |
---|
*** Assignment ***
Topic(s): example1, example2
Partition(s): 0-2 |
If a second workflow is started, an automatic rebalance is triggered, and the first workflow will collect messages from two partitions, and the second from the third partition. The debug output from the collectors of the two workflows will look like this:
Code Block |
---|
*** Rebalance ***
Topic(s): example1, example2
Partition(s): 0-1 |
Code Block |
---|
*** Assignment ***
Topic(s): example1, example2
Partition(s): 2 |
Code Block |
---|
*** Rebalance ***
Topic(s): example1, example2
Partition(s): 0 |
Code Block |
---|
*** Rebalance ***
Topic(s): example1, example2
Partition(s): 1 |
Code Block |
---|
*** Assignment ***
Topic(s): example1, example2
Partition(s): 2 |
If a fourth workflow is started, it will not be assigned any partitions since there are no partition partitions left and there can only be one workflow collecting from one partition. The fourth workflow will act as an active stand-by workflow. If a workflow aborts, a new automatic rebalance is triggered and the active stand-by workflow can be used.
|