This section describes some good-to-know information related to objects used by Functions.
Store vs State
The Script Function uses different objects - Store and State. provides different ways of state management. Two popular options are by interacting with the Store
and State
API.
Use state
during execution for high performance in-memory storage and use store
when the data should be persisted.
When dealing with persistence within a Script Function, a common approach is to use store
/sharedStore
API. This is a fail-safe way of storing the data.
Tip | ||
---|---|---|
| ||
Use the |
A common practice when running a batch stream is to read from the store/sharedStore
for every record, update the value and then write to the previously read record the same key in the store. This results in one read and write operation for every record. Processing a file with 10000 records would result in 20000 store/sharedStore
operations.
Instead, a suggestion would be to use the state
object to store the state in memory and only read and write to the store/sharedStore
when required. The state
object is much faster than store/sharedStore
object.
Example
The following example illustrates how to use state
and store
. We recommend to use state
during transform and store
in flush.
...
1 |
|
Best Practices to Avoid Stateful Stream Design
Real-time streams that utilize the state variable when using the Script Function are at risk of losing data stored in the memory if there is an issue with the execution of the stream. When a stream aborts or fails due to any reason, using the state variable will not help since it is temporary and only available during a single execution of the stream.
...