The following guide can be used to assist you when creating your unique batch scaling solution. An important thing to remember is that you cannot mix standard agents with scaling agents in the same workflows. Workflows with standard agents save the state in Usage Engine and workflows with batch agents save the state in Kafka.
Creating a scalable solution (example)
These are high-level steps to creating a scalable batch solution in Usage Engine. The following example solution is made up of several profiles including the newly created Partition Profile (4.3) and Scalable Inter Workflow Profile (4.3), and two workflow types, Batch Scaling Collection and Batch Scaling Processing.
Create a Partition Profile
Create the Aggregation, Duplicate UDR, and Scalable Inter Workflow profiles and link the Partition Profile created in Step 1 to each.
Create the workflows.
Batch Scaling Collection Workflow
Batch Scaling Processing Workflow(s) - can be one or a series of workflows.
Note!
You can include multiple Aggregation and Duplicate UDR agents within the same workflow. These agents can either share the same Partition Profile or use different Aggregation and Duplicate UDR Profiles. For instance, you might use different profiles if you need to apply a different ID field as the Key in storage.
Decide how many maximum workflows should execute in parallel. Think about how you can evenly distribute your data between workers.
Warning!
Try to pick a Max Scale Factor that is divisible by many other numbers, like 6 or 12. You need to ensure that it is high enough to handle the data coming in, but not so high that you will overload resources.
Finally, you must select an identifier the workflow will use to distribute the UDRs. Typically, this would be a field based on the record group, like a customer ID or an account number. You can also create and populate a field using APL, see https://infozone.atlassian.net/wiki/x/AoB8Fw
Note!
When creating a scalable workflow you need to add the Kafka profile in the execution tab of the workflow properties.
Scaling Batch Workflows
Usage Engine will scale out and in and re-balance scalable batch workflows automatically and you can schedule when to start a scale-out or scale-in.
Deploying a scale-out configuration with ECDs:
Use https://infozone.atlassian.net/wiki/x/IgMkEg with Dynamic Workflows to define how to package a scale-out. See the tabs on https://infozone.atlassian.net/wiki/x/VgQkEg for more information.
You can schedule the ECD and workflow to start or stop at specific times, or, you can start them manually.
Scaling option | Description |
|---|---|
Automatic Scaling | Based on a configured metric, for example, scaling starts when more than X number of UDRs are received for processing and stops when the received UDRs drop below this number. |
Manual Scaling | There are two options for manual scaling;
|