...
When creating a scalable batch workflow in Usage Engine, it’s important to ensure that all agents with storage capabilities are configured to use Kafka storage. Additionally, scalable workflows require scalable Inter Workflow Collection and Forwarding Agents, as regular Inter Workflow Agents are not compatible. Mixing agents with different storage types, such as a Data Aggregator agent configured with Kafka storage and another with file storage, within the same workflow is not supported.
An important thing to remember is that you cannot mix standard agents with scaling agents in the same workflows. Workflows with standard agents save the state in Usage Engine and workflows with batch agents save the state in Kafka.
Creating a scalable solution (example)
These are high-level steps to creating a scalable batch solution in Usage Engine. The following example solution is made up of several profiles including the newly created Partition Profile (4.3) and Scalable Inter Workflow Profile (4.3), and two workflow types, Batch Scaling Collection and Batch Scaling Processing.
Choose Decide on your scaling factor, this will be the maximum number of workflows that can effectively cooperate to process a batch. This is an important choice and will be difficult to change once your workflows are in production.
...