A batch scaling solution requires you to use a combination of new agents and some existing agents that have had configuration setting updates. Three key things to note are:
All storage related to the new agents is in Kafka. This includes transactions and backend working storage.
Workflows containing these new agents can be scaled without any configuration changes. The system will automatically handle the re-balancing of which workflows use which partition.
You cannot mix standard Inter WF agents with scaling Inter WF agents in the same workflow. This is because workflows with standard Inter workflow agents save the state in UsageEngine and workflows with batch agents save the state in Kafka.
Key agents used in scalable workflows
Agent name | Description |
|---|---|
Scalable Inter WF Collection Agent  | This is a newly created agent specifically for batch scaling workflow groups. It functions similarly and has the same configuration as the https://infozone.atlassian.net/wiki/x/fRgkEgagent. The key difference is… |
Scalable Inter WF Forwarding Agent  | This is a newly created agent specifically for batch scaling workflow groups. It functions similarly and has the same configuration as the https://infozone.atlassian.net/wiki/x/uBckEg agent. The key difference is… |
Aggregation Agent  | A new Kafka storage option has been added to the existing Aggregation Agent. When setting up your batch scaling solution, you must add this storage option to the agent configuration. Some use cases may need to switch to the new profile. See, Aggregation Agent(4.3). |
Duplicate UDR Agent  | A new Kafka storage option has been added to the Duplicate UDR Agent. When setting up your batch scaling solution, you must add this storage option in the agent configuration. See Duplicate UDR(4.3). |
Subsections
This section contains the following subsections:
Questions:
Can agents other than those in the table above be used in a scaling WF?
In the aggregation agent description, we advise that some use cases may require the user to switch to the new profile. when would this need to happen?
What is the key difference between the Regular Inter WF Agents and the new agents (is it that the new scalable agents will be able to partition the data? maybe also that they use Kafka topics for storage?)