Page Comparison

Example - Collect Messages from 2 Topics with 3 Configured Partitions

You want to collect messages from 2 Kafka topics called “example1” and “example2” which have 3 configured partitions each.

In the Kafka collection agent configuration, you only need to state the names of the topics and consumer group, and the agent will automatically manage the assignment of partitions within the consumer group.

You will get the following behavior and debug in the executing workflows:

• If only one workflow is started, it will collect messages from all three partitions. The debug output from the collector will look like this:

*** Assignment ***
Topic(s):       example1, example2
Partition(s):   0-2

• If a second workflow is started, an automatic rebalance is triggered, and the first workflow will collect messages from two partitions, and the second from the third partition. The debug output from the collectors of the two workflows will look like this:

*** Rebalance ***
Topic(s):       example1, example2
Partition(s):   0-1

*** Assignment ***
Topic(s):       example1, example2
Partition(s):   2

• If a third workflow is started, an automatic rebalance is triggered, and each workflow will collect messages from one partition. The debug output from the collectors of the three workflows:

*** Rebalance ***
Topic(s):       example1, example2
Partition(s):   0

*** Rebalance ***
Topic(s):       example1, example2
Partition(s):   1

*** Assignment ***
Topic(s):       example1, example2
Partition(s):   2

• If a fourth workflow is started, it will not be assigned any partitions since there are no partition partitions left and there can only be one workflow collecting from one partition. The fourth workflow will act as an active stand-by workflow. If a workflow aborts, a new automatic rebalance is triggered and the active stand-by workflow can be used.