Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

This page contains a collection of examples of how different kind kinds of yaml files for ECDs may look like

ECD for Batch Scheduling

For batch use cases, you typically want to create an ECD without any workflows and then configure the batch scheduling using the Workflow Group scheduling rules. An EC group with the same name as the ECD will automatically be created, which means it is straight forward to start configuring scheduling rules. Note that since Since the ECD is inherently scalable, there is no need to perform additional configuration to distribute the batch scheduling over more machines. Scaling out the ECD using the built-in autoscaler is enough.

ECDs for batch scheduling can be used together with the dynamic workflow REST API to dynamically manage workflows executing on the ECD. For instance, adding workflows to collect from new sources can be done using the /ops/api/v1/workflows/{templateName}/{workflowName} API, with relevant workflow parameters to define the collection point. When enough workflows are added to make the CPU or memory of a single EC over-utilised, the ECD can be scaled out, either automatically using autoscaling with metric threshold, or manually by changing the max-/min-settings of the ECD autoscaler.

...