...
The Storage tab contains settings that are specific for File Storage, Couchbase Storage, Redis Storage, Elasticsearch Storage, and SQL Storage.
Couchbase Storage
...
Setting | Description |
|---|---|
Profile | Select a Couchbase (4.2) profile. This profile is used to access the primary storage for aggregation sessions. |
Mirror Profile | Selecting this Couchbase profile is optional. It is used to access a secondary storage, providing read-only access for aggregation sessions. Typically, the Mirror Profile is identically configured to a (primary) Profile, that is used by workflows on a different EC or other Usage Engine system. This is useful to minimize data loss in various failover scenarios. The read-only sessions can be retrieved with APL commands. For more information and examples, see Aggregation Functions(4.2). |
...
Setting | Description |
|---|---|
Profile | Select an Elasticsearch (4.2) profile. This profile is used to access the storage for aggregation sessions. |
File Storage
...
Setting | Description | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
Storage Host | You can only select Automatic. When you select Automatic, the EC used by the running workflow is automatically applied. Alternatively, if the Aggregation Session Inspector is used, a storage host is selected automatically. For further information, see Aggregation Session Inspector(4.2).
| ||||||||||
Directory | Enter the directory on the Storage Host where you want the aggregation data to be stored. The directory must be a shared file system between all the ECs. | Partial File Count | In this field, you can enter the maximum number of partial files that you want to store. Consider the following: Startup: All the files are read at startup. It takes longer if there are many partial files. Transaction
If this field is greyed out with a stated directory, it means that the directory path has been hard-coded using the
| ||||||||
Partial File Count | In this field, you can enter the maximum number of partial files that you want to store. Consider the following: Startup: All the files are read at startup. It takes longer if there are many partial files. Transaction commitment: Many small files (large Partial File Count) increase performance when the transactions are committed. In a batch workflow, use this variable to tune performance.
| ||||||||||
Max Cached Sessions | Enter the maximum number of sessions to keep in the memory cache. This is a performance-tuning parameter that determines the memory usage of the Aggregation agent. Set this value to be low enough so that there is still enough space for the cache in memory, but not too low, as this will cause performance to deteriorate, see Performance Tuning with File Storage(4.2) for more information. | ||||||||||
Enable Separate Storage Per Workflow | This option enables each workflow to have a separate session storage. Multiple workflows are allowed to run simultaneously using the same Aggregation profile. If this checkbox is selected, a workflow will never see a session from another workflow. |
...
| Note |
|---|
Note!Sometimes, you may notice that file storage takes up more space than expected. This is expected behavior. Read through this note for an overall understanding of the way file storage in Aggregation works. When session data is stored, it is appended to the session file. This means that old session data from the session file is still present in the storage and the current version is added to the file. Removal of old data is done only under certain conditions because otherwise, aggregation handling would be too slow. This is why file storage takes up more space than calculated with session number and single session object size. The session files on the disk grow up to a certain threshold ( 50MB by default) and then a new file is created and used. The old session file will be deleted when no more active sessions are stored in it. The accepted size of a session file can be adjusted by using aggregation.min_session_file_size parameter. For instance, aggregation.min_session_file_size=20000000 will set it to 20MB. Old files are removed during the storage commit. Also, since there is a possibility that there will be old session files present because of some long-lived sessions stored there, a defragmentation algorithm is implemented. It runs occasionally and moves those long-lived sessions to new session files so that old session files can be deleted. This is why aggregation storage takes up a lot of disk space. It is designed to provide higher performance at the expense of higher disk space consumption. |
Memory Only
...
When you have selected Memory Only as storage, there are no additional settings in the Storage tab.
Redis Storage
...
Setting | Description |
|---|---|
Profile | Select a Redis (4.2) profile. This profile is used to access the storage for aggregation sessions. |
...
Setting | Description | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Profile | Select a Database Profile(4.2) profile configured with the SQL database type. This profile is used to access the storage for aggregation sessions.
| ||||||||||||||||
Index Fields | Click the Add button to select the UDR type. | ||||||||||||||||
Table SQL Script | This text box will generate the SQL statements for the selected UDRs' table schema and indexes for Id, TxId. The schema will be generated based on the number of UDRs in the UDR Type Mapping table.
|
...
These fields supports parameterization using ${} syntax, see Profiles(4.2) for more information on how parameterization works.
Couchbase Storage
...
You can also set the properties listed in the Advanced tab as Execution properties. This will override the values that are set in the profile, including default values.
...
Note!
...
syntax, see Profiles(4.2) for more information on how parameterization works.
Couchbase Storage
...
You can also set the properties listed in the Advanced tab as Execution properties. This will override the values that are set in the profile, including default values.
| Info | |||||
|---|---|---|---|---|---|
Example - Overriding the Advanced properties
|
Note!
See the Note at the end of this page for more information when using pessimistic or optimistic locking mechanisms for Couchbase aggregation storage.
Elasticsearch Storage
...
For Elasticsearch storage, you can modify the properties listed as shown above in the Advanced tab.
...
For Redis Storage, you can only modify the properties in the Advanced tab.
...
Note!
...
See the Note at the end of this page for more information when using optimistic locking for Redis aggregation storage.
SQL Storage
...
For SQL Storage, you can modify the properties listed as shown above in the Advanced tab.
...
Note!
When using Couchbase or Redis aggregation storage, it is important to take note of the concept of locking mechanisms when configuring workflows. Locking mechanisms are of two types: Pessimistic and Optimistic.
Redis aggregation storage only has an Optimistic lock whereas Couchbase aggregation storage has both Optimistic and Pessimistic locks.
Pessimistic Lock
When a workflow thread is working on a session, it is considered to be fully locked. No other thread can work on that particular session. Once the first thread is finished, the lock is released and another thread can take the lock and work on the session.Optimistic Lock
Instead of acquiring a traditional lock for a session, a workflow thread obtains a CAS (Compare And Swap) for that session. The CAS serves as a kind of hash or fingerprint of the session data. When the consume block is done and the session is ready to be updated, an error occurs if the CAS no longer matches. In scenarios where multiple threads have made updates to the same session, only the changes from the first thread to complete its work are accepted. Any other thread(s) attempting to update will encounter failure and need to restart their work from the beginning. This process ensures that only changes from one workflow at a time can be committed, akin to the principles of pessimistic locking. It's essential to understand a key distinction: the consume and sessionInit blocks may be invoked multiple times due to the retry mechanism mentioned earlier. As a result, it's advisable to avoid using global variables within the aggregation APL. However, the udrRoute function can be safely utilized within these blocks since it is executed only when the Optimistic lock succeeds. If global variables are necessary, they can be relocated to an analysis agent and updated through the udrRoute function.
...
Note!
Threads may live in multiple processes on multiple machines.