Versions Compared

Version Old Version 1 New Version 2
Changes made by

Felicity Borrelli

Felicity Borrelli

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Parameter

Comment

ID Field

Defines how to match a UDR to a partition.

Max Scale Factor

Number of partitions, which is the same as maximum number of workflows that can execute in parallell.

Info

Note that if any of the parameters needs to be changed, it is considered being a new configuration, and hence starting with empty topics.

If you want to use the existing data, you must use the standard Kafka Agents and migrate the data. Or do we even want to mention this?

Scale out/in Design

PE will scale out and in and re-balance automatically. You can also schedule a scale out (and scale in).

  1. “Packaging” a scale-out configuration:
    Use the regular ECD definition using Dynamic Workflows for defining how to package a scale out. For instance:

    1. Collection WF scales with 1 (one) extra WF per ECD.

    2. Processing WF scales with 3 (three) extra WFs per ECD.

    3. Or combine the above into the same ECD.

  2. Scheduling a scale-out configuration:

    1. Automatic; the system will scale automatically based on metrics.

    2. Manual; schedule the ECD and WF start up (or stop).

Automatic Scaling

Manual Scaling

  • Based on Metric.

  • Should also have some “duration” of the metric to avoid oscillating behaviour?

  • You can start up ECDs manually.

  • We have a way of scheduling ECDs as well.