Data Aggregator
The Data Aggregator Function consolidates related records that originate from either a single or multiple sources, into a single record, and turns it into a more useful format for analysis and decision-making. Related records are grouped into sessions according to the value of their respective fields, and a set of configurable conditions.
As an example, streams using the Data Aggregator Function can help billing systems that have limitations on the number of usage events that can be processed. Using this Function, you can perform aggregation operations such as SUM, COUNT, MAX, MIN, AVERAGE, and many more on any configured field(s), as well as set conditions on those fields when flushing data.
For example, accumulate usage records over time, per account, subscriber, and service to calculate the total usage of a service over a month and then use the aggregated data to deliver to a billing service or to be used as an input for a cloud-based storage service.
Configuration
The Data Aggregator Function's configuration contains the following sections:
Group Fields
Here you specify the names of all the fields you want to group together when performing the aggregation.
Field | Description |
---|---|
Fields | Specify the fields (keys)Â to be used for grouping aggregated data. If a record has the same values as another record in the selected fields, they will be grouped in the same session. ExampleFor example, there are three records being read in a stream with field key, user and you decide to group by user
So, You can either select the fields from the drop-down menu or type the fields one by one into Add field name and click the + button. |
Group based on date/time | Select the checkbox to enable aggregation based on a time period. Specify applicable fields, containing timestamp data, to be used for grouping of aggregation data. Select the Period to group the information by:
Time Period FormatThe supported time period format is ISO 8601 (extended format) in accordance with the YYYY-MM-DDTHH:mm:ss.sssZ pattern. In case you need to convert a date format to support the ISO standard, see Script. An example would be: "2019-01-13T00:00:00.000Z". Click  to add additional grouping criteria based on the time period. |
Configuration
Any change in the Group Fields configuration may result in a new aggregation session, however, the old sessions will still be in the storage. So if you change it back to the previous configuration, the already existing session in the storage (if any) will be used during the aggregation. Refer to the note for more information on the TTL of an aggregation session.
Aggregate Fields
Here you specify the name of the field(s) on which the aggregation operation will be performed:
Field | Operation |
---|---|
Name of the input field key | The Operation drop-down menu is divided into three types:Â
|
Click + Aggregate to add more fields for aggregation.
Flush by
Here you select how and when to flush the aggregated data to the next Function in the stream.
In Usage Engine, you can flush using any of the following options:
Flush by | Description |
---|---|
End of transaction | Aggregated data is flushed at the end of each transaction. This option is not applicable to real-time streams. |
End of stream | Aggregated data is flushed at the end of the stream. This option is not applicable to real-time streams. |
Timeout | Aggregated data is timed out when a predefined interval has passed or a condition is met. In the case of batch streams, the flush for timed out aggregated data happens only when a stream is executed. For real-time streams, the aggregated data is checked every 60 seconds and flushed in case of timeout. Select the type of timeout based on:
Click + Add Condition if you wish to add custom conditions to timeout the aggregated data. This option is available only when the Timeout flush-by option is selected. |
Â
Metadata
You can view and access the following metadata properties of aggregated session. To view the metadata, use the meta object as mentioned in the Script Function. Here is an example:
Property name | Description |
---|---|
count | Number of aggregated records |
flushType | The reason for the session being flushed out. Shows any of the values: ALL_FILES, EACH_FILE, TIMEOUT and CONDITION. During the preview, the value will be empty. |
firstEvent | Date and time of the first aggregated record in the session |
lastEvent | Date and time of the last aggregated record in the session |