Aggregation Profile
- Jenni Wallin
- Kevin Wu
- Former user (Deleted)
Take into account the following behaviors when using the Aggregation profile:
- You can apply an Aggregation profile to any number of workflow configurations.
- Aggregation sessions created in the storage that is specified by the profile can be accessed by multiple active workflows simultaneously.
- When you have selected file storage and are using an Aggregation profile across several workflow configurations, you must consider the read-and-write lock mechanisms that are applied to the stored sessions. For further information about read-and-write locks, see Aggregation Agent Configuration - Batch and Aggregation Agent Configuration - Real-Time.
- The Aggregation profile is loaded when you start a workflow that depends on it. Changes to the profile become effective when you restart the workflow.
Configuration
The contents of the buttons in the menu bar may change depending on which configuration type has been opened in the currently active tab. The Aggregation profile uses the standard buttons that are visible for all configurations, and these are described in Build View.
Session Tab
In the Session tab you can browse and select a Session UDR Type and configure the Storage selection settings.
The Aggregation profile configuration dialog - Session tab
| Setting | Description |
|---|---|
Session UDR Type | Click on the Browse... button and select the Session UDR Type. The Session UDR is defined in Ultra. For further information, see Session UDR Type. |
Storage | Select the type of storage for aggregation sessions. The available settings are File Storage, Couchbase Storage, Redis Storage, Elasticsearch Storage, SQL Storage and Memory Only. File Storage and Memory Only can be used in batch and real-time workflows. Elasticsearch Storage and SQL Storage can only be used in batch workflows. Couchbase Storage and Redis Storage can only be used in real-time workflows. These storage types allow highly available systems with geographic redundancy. The session data that is replicated within the storage is available across workflows, EC Groups, and systems. This serves to minimize data loss in failover scenarios. Note! Data stored in Couchbase or Redis is not available in the Aggregation Session Inspector. |
Association Tab
You use the Association tab to configure rules that are used to match an incoming UDR with a session. Every UDR type requires a set of rules that are processed in a certain order. In most cases, only one rule per incoming UDR type is defined.
You can use a primary expression to filter out UDRs that are candidates for a specific rule. If the UDR is filtered out by the primary expression, it is matched with the existing sessions by using one or several ID fields as a key.
For UDRs with ID Fields matching an existing session, an additional expression may be used to specify additional matching criteria. For example, if dynamic IP addresses are provided to customers based on time intervals, the field that contains the IP address could be used in ID Fields while the actual time could be compared in Additional Expression.
The Aggregation profile configuration dialog - Association tab
| Setting | Description |
|---|---|
UDR Types | Click on the Add button to select a UDR Type in the UDR Internal Format dialog. The selected UDR type will then appear in this field. Each UDR type may have a list of rules attached to it. Selecting the UDR type will display its rules as separate tabs to the right in the Aggregation profile configuration. |
Primary Expression | The Primary Expression is optional. Enter an APL code expression that is going to be evaluated before the ID Fields are evaluated. If the evaluation result is Use the |
ID Fields | Click on the Add button to select additional ID Fields in the ID Fields dialog. These fields, along with the Additional Expression settings enable the system to determine whether a UDR belongs to an existing session or not. If the contents of the selected fields match the contents of a session, and an Additional Expression evaluation results in Note! Make sure that the selected fields are of the same type and appear in the same order for all the rules that are defined for the agent. |
Additional Expression | The Additional Expression is optional. Enter an APL code expression that is going to be evaluated along with the ID Fields. Use the The Additional Expression is useful when you have several UDR types with a varying number of ID Fields, that are about to be consolidated. Having several UDR types requires the ID fields to be equal in number and type. If one of the types requires additional fields that do not have any counterpart in the other type or types, these must be evaluated in the Additional Expression field. Save the field contents as a session variable, and compare the new UDRs with it. For an example, see Association - Radius UDRs in Aggregation Example - Association of IP Data. Note! When using Additional Expressions for Aggregation the caching mechanism only takes into account the primary and secondary rules when creating the session CRC. This means that if the number of sessions that cannot be told apart without the use of an Additional Expression is high, the performance of the Aggregation Agent decreases due to cache read/write operations. This is especially true if the Max Cached Sessions property is low compared to the number of sessions. For this reason, it is recommended that Max Cached Sessions is set to a high value when using Additional Expressions. |
Create Session on Failure | Select this check box to create a new session if no matching session is found. If the check box is not selected, a new session will not be created when no matching session is found. Note! If you provide a primary expression, and it evaluates to If the order of the input UDRs is not important, all the rules should have this check box checked. This means that the session object is going to be created regardless of the order in which the UDRs arrive. However, if the UDRs are expected to arrive in a particular sequence, Create Session on Failure must only be selected for the UDR type or field that is considered to be the master UDR, i e the UDR that marks the beginning of the sequence. In this case, all the slave UDR types or fields are targeted for error handling if they arrive before their master UDR. Note! At least one of all defined rules must have this check box selected. Otherwise, no session will ever be created. For further information about all available system properties, see System Properties. |
Add Rule | Click this button to add a new rule for the selected UDR Type. The rule will appear as a new folder to the right of the UDR Types in the Aggregation profile configuration. Usually, only one rule is required. However, in a situation where a session is based on an IP number, stored in either a target or source IP field, two rules are required. The source IP field can be listed in the ID Fields of the first rule and the target IP field listed in the ID Fields of the second rule. |
Remove Rule | Click this button to remove the currently displayed rule. |
Storage Tab
The Storage tab contains settings that are specific for File Storage, Couchbase, Redis, Elasticsearch, and SQL.
File Storage
The Aggregation profile configuration dialog - File Storage settings
| Setting | Description |
|---|---|
Storage Host | Select a Storage Host from the drop-down list. For storage of aggregation sessions select either a specific EC Group or Automatic. If you select Automatic, the same EC Group that has been used by the running workflow will be applied. Alternatively, if the Aggregation Session Inspector is used, a storage host is selected automatically. Refer to Aggregation Session Inspector for further information on the Aggregation Session Inspector. Note! It is recommended that you configure the aggregation workflow to run on the same EC Group that you have selected as Storage Host. |
Directory | Enter the directory on the Storage Host where you want the aggregation data to be stored. Note! If the Storage Host above, is configured to be Automatic, the corresponding Directory has to be a shared file system between all the EC Groups. If this field is greyed out with a stated directory, it means that the directory path has been hard-coded using the Example - Using the mz.preset.aggregation.storage.path property To enable the property and state the directory to be used: mzsh topo set val:common.mz.preset.aggregation.storage.path '/mydirectory/agg'
mzsh topo unset val:common.mz.preset.aggregation.storage.path |
Partial File Count | In this field, you can enter the maximum number of partial files that you want to store. Consider the following: Startup: All the files are read at startup. It takes longer if there are many partial files. Transaction commitment: When the transactions are committed, many small files (large Partial File Count) increase performance. In a batch workflow, use this variable to tune performance. Note! In a real-time workflow, updates to sessions are saved on disk only if the Storage tab is configured with Storage Commit Conditions. |
Max Cached Sessions | Enter the maximum number of sessions to keep in the memory cache. This is a performance-tuning parameter that determines the memory usage of the Aggregation agent. Set this value to be low enough so that there is still enough space for the cache in memory, but not too low, as this will cause performance to deteriorate. For further information see the section below, Performance Tuning with File Storage. |
| Enable Separate Storage Per Workflow | This option enables each workflow to have a separate storage checked for duplicates. Multiple workflows are allowed to run simultaneously using the same Aggregation profile. However, if this checkbox is selected, a UDR in a workflow will not be checked against UDRs in a different workflow. |
Note!
Sometimes, you may notice that file storage takes up more space than expected. This is expected behavior. Read through this note for an overall understanding of the way file storage in Aggregation works.
When session data is stored, it is appended to the session file. This means that old session data from the session file is still present in the storage and the current version is added to the file. Removal of old data is done only under certain conditions because otherwise, aggregation handling would be too slow. This is why file storage takes up more space than calculated with session number and single session object size.
The session files on the disk grow up to a certain threshold ( 50MB by default) and then a new file is created and used. The old session file will be deleted when no more active sessions are stored in it. The accepted size of a session file can be adjusted by using aggregation.min_session_file_size parameter. For instance, aggregation.min_session_file_size=20000000 will set it to 20MB. This parameter is set with the mzsh topo command on EC, cell, or container level.
Old files are removed during the storage commit. Also, since there is a possibility that there will be old session files present because of some long-lived sessions stored there, a defragmentation algorithm is implemented. It runs occasionally and moves those long-lived sessions to new session files so that old session files can be deleted.
This is why aggregation storage takes up a lot of disk space. It is designed to provide higher performance at the expense of higher disk space consumption.
Couchbase Storage
The Aggregation profile configuration dialog - Couchbase Storage
| Setting | Description |
|---|---|
Profile | Select a Couchbase profile. This profile is used to access the primary storage for aggregation sessions. |
Mirror Profile | Selecting this Couchbase profile is optional. It is used to access secondary storage, providing read-only access for aggregation sessions. Typically, the Mirror Profile is identically configured to a (primary) Profile, that is used by workflows on a different EC Group or other |
Mirror profile concept
Elasticsearch Storage
Aggregation profile configuration dialog - Elasticsearch Storage
| Setting | Description |
|---|---|
| Elasticsearch | Select an Elasticsearch profile. This profile is used to access the storage for aggregation sessions. |
Redis Storage
Aggregation profile configuration dialog - Redis Storage
| Setting | Description |
|---|---|
Profile | Select a Redis profile. This profile is used to access the storage for aggregation sessions. |
SQL Storage
Aggregation profile configuration dialog - SQL Storage
| Setting | Description | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Profile | Select an SQL profile. This profile is used to access the storage for aggregation sessions. Note! Currently the SQL storage only supports PostgreSQL and SAP HANA databases. Storage sharing functionality is currently not supported. | ||||||||||||
| Index Fields | Click the Add | ||||||||||||
| Table SQL Script | This text box will generate the SQL statements for the selected UDRs' table schema and indexes for Id, TxId. The schema will be generated based on the number of UDRs in the UDR Type Mapping table. Info! Users will have to copy the SQL script generated in the text box to create the PostgreSQL and SAP HANA tables on their own in the database listed in the Database profile. The Aggregation profile will not automatically create the tables for you. Note! The following table columns are mandatory when creating the database:
|
Advanced Tab
The Advanced tab is available when you have selected Couchbase Storage, Redis Storage, Elasticsearch Storage or SQL Storage in the Session tab. It contains properties that can be used for performance tuning. For information about performance tuning, see Aggregation Performance Tuning.
Couchbase Storage
The Aggregation profile configuration dialog - Advanced tab for Couchbase
You can also set the properties listed in the Advanced tab as Execution properties in the STR. This will override the values that are set in the profile, including default values.
Example - Overriding the Advanced properties
$ mzsh topo set topo://container:container1/pico:ec1/val: \ config.properties.mz.cb.agg.json_serializer.format MZ-BIN
Elasticsearch Storage
The Aggregation profile configuration dialog - Advanced tab for Elasticsearch
For Elasticsearch storage, you can modify the properties listed as shown above in the Advanced tab.
Redis Storage
The Aggregation profile configuration dialog - Advanced tab for Redis
For Redis Storage, you can only modify the properties in the Advanced tab.
SQL Storage
The Aggregation profile configuration dialog - Advanced tab for SQL
For SQL storage, you can modify the properties listed as shown above in the Advanced tab.