...
This example shows a batch processing setup where you collect files and perform duplication checks and aggregation. We want to make this solution scalable to improve the processing times of our data during periods of high usage. We have set up two workflows in our batch scaling solution.
...
The In the File collection workflow the Scalable InterWF Forwarding agent in the File collection workflow manages the partitions. It uses one or more unique ID Fields (e.g. customer ID) to determine which partition a UDR belongs to.
The maximum number of partitions created is determined by the Max Scale Factor parameter in the Partition Profile.
Note!
The number of partitions will be the same across all topics. The points of storage will occur, for example,
With the passing of UDRs between workflows.
When duplicate UDR keys are detected.
For aggregated sessions.
The In the Processing workflow the Duplicate UDR agent will check for duplicates across all partitions. Checked UDRs are placed in an additional topic with the same partitions as the corresponding Collection collection workflow topic. Any duplicate keys are saved in a separate topic.
The Also in the Processing workflow, the Aggregation Agent will collect data from an inter-workflow topic and use a separate aggregation session storage topic.
...