Versions Compared

Old Version 18

changes.mady.by.user Felicity Borrelli

Saved on

compared with

New Version 19

changes.mady.by.user Felicity Borrelli

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Configuration field

Description

Group fields

Note!

Changing how you group data (updating the Group fields) will start new aggregation sessions and the the old sessions will be stored. If you want to revert to the original way of grouping, the system will reuse the old stored sessions, but only if they haven’t timed out yet. See the note on TTL above.

Changing the Group Fields configuration can create a new aggregation session, while previous sessions will remain stored. If you revert to the original configuration, any existing sessions in storage will only be used for aggregation if the TTL (Time to Live) has not expired.

Fields

Specify which fields will be used to group records together for aggregation. If two or more records have the same values in the fields you select, they will be grouped into the same session for processing. You must configure at least one field for the data aggregator to work properly. If left empty, you will receive an error message advising of this.

Info

Example - Grouping user records by the user field

Let’s say you have three records, and you decide to group them by the user field:

  • Record 1 has the user field set to A

  • Record 2 has the user field set to B

  • Record 3 has the user field set to A

In this case, Record 1 and Record 3 will be grouped because they have the same user value (A), while Record 2 will be in a separate group with its own user value (B). If you’re using a COUNT operation, for example, the total for the group with user A will be 2 and the total for the group with user B will be 1

You can either type the field names manually or select them from the drop-down menu.

Group based on date/time

Select the checkbox to specify a defined period for aggregation.

Note!

The supported format is ISO 8601 (extended format), following the pattern YYYY-MM-DDTHH:mm:ss.sssZ.

  • YYYY-MM-DD represents the date (year, month, day).

  • THH:mm:ss indicates the time (hours, minutes, seconds).

  • .sss denotes milliseconds.

  • Z signifies the time is in UTC (Coordinated Universal Time).

For example: "2019-01-13T00:00:00.000Z".

If you need to convert a date format to support the ISO standard, see Script.

Click + Add period field to add additional period fields

Field

If you select to group the fields by date/ time you will specify each field and then select a time duration for each.

Period

Select a time duration from the dropdown:

  • Hour

  • Day

  • Month

  • Year

Info

Example - Aggregating data hourly

Let’s say you configure the Period field to group data by 1 hour, but you set the start time to 12:15 PM instead of the top of the hour.

  • Record 1 arrives at 12:20 - the session gets created and it will collect records until 13:20

  • Record 2 arrives at 12:50 - this record will be collected in the first session.

  • Record 3 arrives at 13:30 - this record will be collected in a new session because it occurs after 13:20.

In this case, Record 1 and Record 2 will be grouped together because they fall within the 1-hour period starting at 12:20. Record 3 will be placed in the next group, which starts at 13:20. The data will be processed and output at the end of each period, based on the custom start time you configured.

Aggregate fields

Field

Specify the name of the field(s) on which the aggregation operation will be performed:

Operation

Select the aggregation operation from the drop-down menu, this operation will apply to the chosen field. The available operations are grouped into two categories: Numeric, and General. Click + Add Aggregate field to add more fields.

Numeric:

  • SUM: Adds up all the numeric values.

  • MAX: Returns the highest numeric value.

  • MIN: Returns the lowest numeric value.

  • AVERAGE: Calculates the mean of the numeric values.

General:

  • COUNT: Counts the number of records.

  • CARRY_FIRST: Uses the first value encountered.

  • CARRY_LAST: Uses the last value encountered.

Flush by

Flush by

Select how and when to flush the aggregated data to the next function in the stream. The options are:

  • End of transaction

  • End of stream

  • Timeout


See below for more information on these options and how Flush by works.

...