Versions Compared

Old Version 23

changes.mady.by.user Felicity Borrelli

Saved on

compared with

New Version 24

changes.mady.by.user Felicity Borrelli

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Configuration field

Description

Group fields

Note!

Changing how you group data (updating the Group fields) will start new aggregation sessions and the the old sessions will be stored. If you want to revert to the original way of grouping, the system will reuse the old stored sessions, but only if they haven’t timed out yet. See the note on TTL above.

Fields

Specify which fields will be used to group records together for aggregation. If two or more records have the same values in the fields you select, they will be grouped into the same session for processing. You must configure at least one field for the data aggregator to work properly. If left empty, you will receive an error message advising of this.

Info

Example - Grouping user records by the user field

Let’s say you have three records, and you decide to group them by the user field:

  • Record 1 has the user field set to A

  • Record 2 has the user field set to B

  • Record 3 has the user field set to A

In this case, Record 1 and Record 3 will be grouped in a session because they have the same user value (A), while Record 2 will be in a separate session with its own user value (B). If you’re using a COUNT operation, for example, the total for the group with user A will be 2 and the total for the group with user B will be 1

You can either type the field names manually or select them from the drop-down menu.

Group based on date/time

Select the checkbox to specify a defined period for aggregation.

Note!

The supported format is ISO 8601 (extended format), following the pattern YYYY-MM-DDTHH:mm:ss.sssZ.

  • YYYY-MM-DD represents the date (year, month, day).

  • THH:mm:ss indicates the time (hours, minutes, seconds).

  • .sss denotes milliseconds.

  • Z signifies the time is in UTC (Coordinated Universal Time).

For example: "2019-01-13T00:00:00.000Z".

If you need to convert a date format to support the ISO standard, see Script.

Click + Add period field to add additional period fields

Field

If you select to group the fields by date/ time you will specify each field and then select a time duration for each.

Period

Select a time duration from the dropdown:

  • Hour

  • Day

  • Month

  • Year

Aggregate fields

Field

Specify the name of the field(s) on which the aggregation operation will be performed:

Operation

Select the aggregation operation from the drop-down menu, this operation will apply to the chosen field. The available operations are grouped into two categories: Numeric, and General. Click + Add Aggregate field to add more fields.

Numeric:

  • SUM: Adds up all the numeric values.

  • MAX: Returns the highest numeric value.

  • MIN: Returns the lowest numeric value.

  • AVERAGE: Calculates the mean of the numeric values.

General:

  • COUNT: Counts the number of records.

  • CARRY_FIRST: Uses the first value encountered.

  • CARRY_LAST: Uses the last value encountered.

Flush by

Flush by

Select how and when to flush the aggregated data to the next function in the stream. The options are:

  • End of transaction

  • End of stream

  • Timeout


See below for more information on these options and how Flush by works.

...

Info

Example- Timeout type set to ‘month’

For example, if If you set the timeout to happen on the 1st of every month at 23:59 in the "UTC" timezone, the system will flush all aggregated data at that exact time, on the 1st of every month.

...

Info

Example - Timeout type set to ‘Based on timestamp field’

If your data has a field containing a timestamp (e.g., "event_time": "2024-09-18T15:00:00"), You can select to flush the data "Based on timestamp field". The system will flush the data according to this timestamp.W

Adding custom conditions

You can add custom conditions when the flush by is set to timeout by clicking + Add condition.

...