...
Always start with a Batch Scaling Collection Workflow that collects from the original file source and forwards UDRs to Kafka.
The Batch Scaling Processing Workflows can be one or a series of workflows. Batch Duplication Check and Aggregation can be part of the same workflow. There can only be ONE Aggregation Agent and ONE Duplication Agent per workflow.
Decide how many maximum workflows that can execute in parallellparallel, i.e. how can you efficiently shard your data in an efficient way / (distribute it evenly) in different groups. Then you need to decide how to decide which identifier / sorting parameter the workflow should use to distribute the UDR. Typically a field based on record group / ID / number etc. If there is no such field, use APL to create and populate such a field (round-robin among shards for instance).
UI Parameters
...
Parameter | Comment |
|---|---|
ID Field | Defines how to match a UDR to a partition. |
Max Scale Factor | Number of partitions, which is the same as maximum number of workflows that can execute in parallell. Note! If any of the parameters neds to be changed, it is considered a new configuration, and they need to start with empty topics. You can use the existing data, but you must use the standard Kafka Agents and migrate the data. Or do we even want to mention this? |
...
Scaling design
PE will scale out and in and re-balance automatically. You can also schedule a scale-out (and scale-in).
“Packaging” a scale-out configuration:
Use the regular ECD (Execution context deployment) definition using Dynamic Workflows for defining to define how to package a scale-out. For instance:A Collection WF Workflow scales with 1 (one) extra WF Workflow per ECD.
A Processing WF Workflow scales with 3 (three) extra WFs Workflows per ECD.
Or combine the above into the same ECD.
Scheduling a scale-out configuration:
Automatic; the system will scale automatically based on metrics.
Manual; schedule the ECD and WF Workflow start up (or stop).
Automatic Scaling | Manual Scaling |
|---|---|
|
|