Aggregation (4.1)
- Jenni Wallin
- Manuel Pereira
- Mattias Lundström
You can apply an Aggregation profile to any number of workflow configurations.Â
Preparations for your Profile
Each Aggregation profile stores sessions of a specific Session UDR type that you define in ultra. This means that your Aggregation profile configuration must include a session UDR type. See the example below:
Example - Defining a Session UDR type in an Ultra Configuration
session SessionUDRType {
int intField;
string strField;
list<drudr> udrList;
};It is recommended that you keep the session UDR as small as possible. A larger UDR decreases performance compared to a small one.
Note!
Take particular care when updating the Ultra formats. It is not possible to collect data from the Aggregation session storage if the corresponding UDR has been renamed. However, if you change the format definition, you can still collect the data.
Changes to the formats are handled as follows:
Default values are assigned to fields that are added or renamed.
Fields that have been removed are ignored.
Default values are assigned to fields with data types that have been changed.
For further information on Ultra formats, see Ultra Format(4.1).
Profile Configuration
To create an Aggregation profile, go to the Configuration icon and select the profile from the list.
The profile consists of four tabs:
Session Tab
Association Tab
Storage Tab
Advanced Tab
Session Tab
In the Session tab you can browse to and select a Session UDR Type and configure the Storage selection settings.
Setting | Description |
|---|---|
Session UDR Type | Click Browse to search for the Session UDR type that you want to use. See Preparations for your Profile above for information on creating a Session UDR type. |
Storage | Select the type of storage for aggregation sessions. The available settings are Couchbase Storage, Elasticsearch Storage, File Storage, Memory Only, Redis Storage and SQL Storage.
Note! Data stored in Couchbase or Redis is not available in the Aggregation Session Inspector(4.1). |
Association Tab
You use the Association tab to configure rules that are used to match an incoming UDR with a session. Every UDR type requires a set of rules that are processed in a certain order. In most cases, only one rule per incoming UDR type is defined.
Setting | Description |
|---|---|
UDR Types | Click the Add button to select a UDR type in the UDR Internal Format dialog. The UDR type that you select then appears in this field. A UDR type may have a list of rules attached to it. When you select the UDR type, its rules appear as separate tabs to the right in the Aggregation profile configuration. |
Primary Expression | The Primary Expression is optional. You can enter an APL code expression that is to be evaluated before the ID Fields are evaluated. If the evaluation result is Use the |
ID Fields | Click the Add button to select additional ID Fields in the ID Fields dialog. These fields, along with the Additional Expression settings, enable Usage Engine to determine whether a UDR belongs to an existing session or not. If the contents of the selected fields match the contents of a session and an Additional Expression  evaluation results in |
Additional Expression | The Additional Expression is optional. Enter an APL code expression that is to be evaluated with the ID Fields. Use the This setting is useful when you have several UDR types with a varying number of ID Fields to be consolidated. Having several UDR types requires the ID fields to be equal in number and type. If one of the types requires additional fields that do not have any counterpart in the other type or types, these must be evaluated in the Additional Expression field. Save the field contents as a session variable, and compare the new UDRs with it. |
Create Session on Failure | Select this check box to create a new session if no matching session is found. If the check box is not selected, a new session is not created when no matching session is found. If the order of the input UDRs is unimportant, select this check box for all the rules. This means that the session object is created regardless of the order in which the UDRs arrive. However, if the UDRs are expected to arrive in a particular sequence, only select Create Session on Failure for the UDR type/field that is considered to be the master UDR, i.e. the UDR that marks the beginning of the sequence. In this case, all the slave UDR types/fields are targeted for error handling if they arrive before their master UDR. |
Add Rule | Click this button to add a new rule for the selected UDR type. The rule appears as a new folder to the right of the UDR types in the Aggregation profile configuration. Usually, only one rule is required. However, in a situation where a session is based on an IP number, stored in either a target or source IP field, two rules are required. The source IP field can be listed in the ID Fields of the first rule and the target IP field listed in the ID Fields of the second rule. |
Remove Rule | Click this button to remove the selected rule. |
Storage Tab
The Storage tab contains settings that are specific to the selected storage.
Couchbase Storage
Setting | Description |
|---|---|
Profile | Select a Couchbase (4.1) profile. This profile is used to access the primary storage for aggregation sessions. |
Mirror Profile | Selecting this Couchbase profile is optional. It is used to access a secondary storage, providing read-only access for aggregation sessions. Typically, the Mirror Profile is identically configured to a (primary) Profile, that is used by workflows on a different EC or other Usage Engine system. This is useful to minimize data loss in various failover scenarios. The read-only sessions can be retrieved with APL commands. For more information and examples, see Aggregation Functions(4.1). |
Â
Elasticsearch Storage
Setting | Description |
|---|---|
Profile | Select an Elasticsearch (4.1) profile. This profile is used to access the storage for aggregation sessions. |
File Storage
Setting | Description |
|---|---|
Storage Host | You can only select Automatic. When you select Automatic, the EC used by the running workflow is automatically applied. Alternatively, if the Aggregation Session Inspector is used, a storage host is selected automatically. For further information, see Aggregation Session Inspector(4.1). |
Directory | Enter the directory on the Storage Host where you want the aggregation data to be stored. The directory must be a shared file system between all the ECs. |
Partial File Count | In this field, you can enter the maximum number of partial files that you want to store. Consider the following: Startup: All the files are read at startup. It takes longer if there are many partial files. Transaction commitment: Many small files (large Partial File Count) increase performance when the transactions are committed. In a batch workflow, use this variable to tune performance. |
Max Cached Sessions | Enter the maximum number of sessions to keep in the memory cache. This is a performance-tuning parameter that determines the memory usage of the Aggregation agent. Set this value to be low enough so that there is still enough space for the cache in memory, but not too low, as this will cause performance to deteriorate. |
Enable Separate Storage Per Workflow | This option enables each workflow to have a separate storage checked for duplicates. Multiple workflows are allowed to run simultaneously using the same Aggregation profile. However, if this checkbox is selected, a UDR in a workflow will not be checked against UDRs in a different workflow. |
Memory Only
Redis Storage
Setting | Description |
|---|---|
Profile | Select a Redis (4.1) profile. This profile is used to access the storage for aggregation sessions. |
SQL Storage
Setting | Description |
|---|---|
Profile | Select a Database Profile(4.1) profile. This profile is used to access the storage for aggregation sessions. |
Index Fields | Click the Add button to select the UDR type. |
Table SQL Script | This text box will generate the SQL statements for the selected UDRs' table schema and indexes for Id, TxId. The schema will be generated based on the number of UDRs in the UDR Type Mapping table. |
Advanced Tab
The Advanced tab is available when you have selected Couchbase Storage, Elasticsearch Storage, Redis Storage or SQL Storage in the Session tab. It contains properties that can be used for performance tuning. For information about performance tuning, see Aggregation Performance Tuning(4.1).
These fields supports parameterization using ${} syntax, see Profiles(4.1) for more information on how parameterization works.
Couchbase Storage
You can also set the properties listed in the Advanced tab as system properties for the ECD, see Creating an EC Deployment (4.1). This will override the values that are set in the profile, including default values.
Elasticsearch Storage
For Elasticsearch storage, you can modify the properties listed as shown above in the Advanced tab.
Redis Storage
For Redis Storage, you can only modify the properties in the Advanced tab.
SQL Storage
For SQL Storage, you can modify the properties listed as shown above in the Advanced tab.