Configuration

Introduction.. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur accumsan malesuada leo sed fermentum. Nullam non vehicula ligula, ut facilisis nisl. Curabitur at iaculis nisl, sit amet luctus justo. Cras hendrerit quam orci, eget consectetur eros congue eu. Aliquam pulvinar tempor mi, quis efficitur quam cursus a.

Include an image of an example workflow..

Workflow design guide

Always start with a Batch Scaling Collection Workflow that collects from the original file source and forwards UDRs to Kafka.
The Batch Scaling Processing Workflows can be one or a series of workflows. Batch Duplication Check and Aggregation can be part of the same workflow. There can only be ONE Aggregation Agent and ONE Duplication Agent per workflow.
Decide how many maximum workflows that can execute in parallell, i.e. how can you shard your data in an efficient way / distribute evenly in different groups. Then you need to decide how to decide which identifier / sorting parameter the workflow should use to distribute the UDR. Typically a field based on record group / ID / number etc. If there is no such field, use APL to create and populate such a field (round-robin among shards for instance). UI Parameters:

Parameter	Comment
ID Field	Defines how to match a UDR to a partition.
Max Scale Factor	Number of partitions, which is the same as maximum number of workflows that can execute in parallell. Note that if any of the parameters needs to be changed, it is considered being a new configuration, and hence starting with empty topics. If you want to use the existing data, you must use the standard Kafka Agents and migrate the data. Or do we even want to mention this?

Scale out/in Design

PE will scale out and in and re-balance automatically. You can also schedule a scale out (and scale in).

“Packaging” a scale-out configuration:
Use the regular ECD definition using Dynamic Workflows for defining how to package a scale out. For instance:
1. Collection WF scales with 1 (one) extra WF per ECD.
2. Processing WF scales with 3 (three) extra WFs per ECD.
3. Or combine the above into the same ECD.
Scheduling a scale-out configuration:
1. Automatic; the system will scale automatically based on metrics.
2. Manual; schedule the ECD and WF start up (or stop).

Automatic Scaling	Manual Scaling
Based on Metric. Should also have some “duration” of the metric to avoid oscillating behaviour?	You can start up ECDs manually. We have a way of scheduling ECDs as well.