Shared store usage

Shared store is a storage layer that allows streams within the same solution to persist and share data between executions. For example, if Stream A writes a value to shared store, Stream B can later read it.

The store behaves like a Key-Value database:

• Each piece of data is associated with a unique key.

• Data can only be accessed by knowing its corresponding key.

The shared store service is accessed via the variable, sharedStore, that can be used in functions where you can add custom script, like:

• Script

• Script aggregator

See Script aggregator examples for examples of sharedStore usage.

When a stream writes to the shared store, the data is first held in memory (cached) and not immediately visible to other streams. It becomes available only after it is persisted, this happens according to a checkpointing mechanism:

• Scheduled streams persist data at the end of each transaction.

Until persistence occurs, other streams cannot read the newly written data.

When a stream starts, it loads all keys and values from the shared store into memory, called the cached state. This in-memory copy remains unchanged throughout the stream's execution unless the stream writes new data to the shared store or restarts. Any updates to the shared store made by other streams are not visible until the next execution.

When setting a key/value pair, you can configure a TTL (Time-To-Live). The TTL defines how long the value remains in the store before it’s automatically deleted.

By default, the TTL is 30 days, meaning the key will expire 30 days after it’s written or last updated. Each time you write to the same key, its TTL is refreshed. See the sharedStore variable on Script aggregator examples for syntax examples of setting a TTL.

Because the shared store relies on in-memory caching during stream execution, excessive or long-lived data can lead to high memory consumption. There is no fixed limit on the total number of key-value pairs you can store. However:

• Each key/value pair can be up to 16 MB in size. Larger values won’t be stored.

• Because shared store data is loaded into memory when a stream starts, the amount of memory used grows with the size of the stored data.

To help the shared store run smoothly:

• Keep stored data as compact as possible.

• Avoid storing large numbers of keys or large values, especially in memory-constrained environments.