...
The following guide can be used to assist you when creating your unique batch scaling solution. An important thing to remember is that you cannot mix standard agents with scaling agents in the same workflows. Workflows with standard agents save the state in Usage Engine and workflows with batch agents save the state in Kafka.
| Info |
|---|
notes from Michal convo! Cannot have input from regular IWF collector and expect it to scale. - you can mix agents in WFs. can mix all agents that don’t have storage options - but cannot mix agents that have storage capability i.e. 1 agg agent with kafka storage and 1 agg agent with file storage. |
Creating a scalable solution (example)
...
You must choose one or more fields in your UDRs that will be used to partition data. These fields may be based on a record group like a customer ID or an account number.
Create a Kafka Profile pointing to your cluster
Create a Partition Profile where you define your Max Scale Factor and your partitioning fields.
Create the Aggregation, Duplicate UDR, and Scalable Inter Workflow profiles and link the Partition Profile created in Step 2 to each.
Create your workflows.
Standard workflows to prep - prepare data for scaling by sending it to the scalable InterWF Forwarder
Scalable processing workflows start - collect data with a Scalable InterWF Collector.
...
| Note |
|---|
Warning! |
...
must add the Kafka profile in the execution tab of the workflow properties. |
Note!
You can include multiple Aggregation and Duplicate UDR agents within the same workflow. These agents can either share the same Partition Profile or use different Aggregation and Duplicate UDR Profiles. For instance, you might use different profiles if you need to apply a different ID field as the Key in storage.
...
Use https://infozone.atlassian.net/wiki/x/IgMkEg with Dynamic Workflows to define how to package a scale-out. See the tabs on https://infozone.atlassian.net/wiki/x/VgQkEg for more information.
You can schedule the ECD and workflow to start or stop at specific times, or, you can start them manually.
...
Scaling option
...
Description
...
Automatic Scaling
...
Based on a configured metric, for example, scaling starts when more than X number of UDRs are received for processing and stops when the received UDRs drop below this number.
...
Manual Scaling
There are two options for manual scaling;
...
Manually start the ECD and Workflow when you want them to begin.
...
.