...
Include an image of an example workflow..
Note!
Note that you cannot mix standard agents with scaling agents in the same workflow. Workflows with standard agents save the state in PE. Workflows with batch agents save the state in Kafka.
Workflow design guide
Always start with a Batch Scaling Collection Workflow that collects from the original file source and forwards UDRs to Kafka.
The Batch Scaling Processing Workflows can be one or a series of workflows. Batch Duplication Check and Aggregation can be part of the same workflow. There can only be ONE Aggregation Agent and ONE Duplication Agent per workflow.
Decide how many maximum workflows that can execute in parallel, i.e. how can you efficiently shard your data (distribute it evenly) in different groups. Then you need to decide how to decide which identifier / sorting parameter the workflow should use to distribute the UDR. Typically a field based on record group / ID / number etc. If there is no such field, use APL to create and populate such a field (round-robin among shards for instance).
...