Versions Compared

Version Old Version 13 New Version 14
Changes made by

Felicity Borrelli

Felicity Borrelli

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Are there any prerequisites required to be able to configure automatic batch scaling?…

How it works

Assume that you have a batch use case where you collect files, and have to do duplication checks and aggregation. You want to be able to scale. You need 2 or 3 WFs. In the picture below we use 3 WFs.

You collect a large number of files and you want to process the data in them more efficiently. This can be achieved by creating… , maybe number these…

  1. Firstly, the File collection workflow(s) will use an ID Field (e.g. customer ID) to determine which shard/partition a UDR belongs to. - they manage the InterWF partitions

...

Info

From Chat GPT re: Topics - For draft purposes only:

In a software context, especially in messaging and streaming platforms like Kafka, a topic isn’t a type of storage in the traditional sense, like a cache or database. Instead, it refers to a "channel" or "feed" where messages (like UDRs) are grouped and published for consumers to read from. While a topic involves data persistence (messages are stored temporarily or longer-term, depending on configuration), it's more about organizing and transmitting data rather than being a storage unit itself.

In comparison, a cache is a direct storage solution intended for fast access to data. Topics, on the other hand, are about managing and distributing data streams efficiently across systems.

Assume that you have a batch use case where you collect files, and have to do duplication checks and aggregation. You want to be able to scale. You need 2 or 3 WFs. In the picture below we use 3 WFs

.

Subsections

This section contains the following subsections:

...