Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This section describes some good-to-know information related to objects used by  Functions.

Store vs State

The  Script  Function uses different objects -  Store and State. provides different ways of state management. Two popular options are by interacting with the Store and State API.

Use state during execution for high performance in-memory storage and use store when the data should be persisted.

When dealing with persistence within a Script Function, a common approach is to use store/sharedStoreAPI. This is a fail-safe way of storing the data.

Tip
titleRecommendation

Use the store/sharedStore with care and not too frequently as it could have a great impact on performance.

A common practice when running a batch stream is to read from the store/sharedStore for every record, update the value and then write to the previously read record the same key in the store. This results in one read and write operation for every record. Processing a file with 10000 records would result in 20000 store/sharedStore operations.
Instead, a suggestion would be to use the state object to store the state in memory and only read and write to the store/sharedStore when required. The state object is much faster than store/sharedStore object.

Example

The following example illustrates how to use state and store. We recommend to use state during transform and store in flush. 

...

1
2
3
4
5
6
7
8
9
10
11

// transform
let sum = await store.get('sum');
if (sum === undefined) {
  sum = 0;
}
 
sum += payload.value;
 
await store.set('sum', sum);
 
await push({ sum });

Best Practices to Avoid Stateful Stream Design

Real-time streams that utilize the state variable when using the Script Function are at risk of losing data stored in the memory if there is an issue with the execution of the stream. When a stream aborts or fails due to any reason, using the state variable will not help since it is temporary and only available during a single execution of the stream. 

...