Deduplicate input/output data

This page provides examples of what your input and output data might look like in your stream’s Deduplicate function after your stream has been run at least once.

Here are some example streams that use the Deduplicate function. These pre-configured streams can be found and executed from the UsageCloud platform for further learning. See https://infozone.atlassian.net/wiki/x/rMB3 for more details.

In the https://infozone.atlassian.net/wiki/spaces/DAZ/pages/7842352 stream, the Deduplicate function is configured to check every column and discard any duplicates. The records are discarded from the cache after a day.

The configuration of the Deduplicate function in the Handling erroneous and duplicated records stream

Input data

Below is a sample of the input data of the Deduplicate function in the https://infozone.atlassian.net/wiki/spaces/DAZ/pages/7842352 stream which has duplicated records from the Simulate data (https://infozone.atlassian.net/wiki/spaces/DAZ/pages/868843521) function that is simulating the data.

Input data in the Deduplicate function with duplicate records

[
  {
    "date": "2025-07-11T06:09:41.332Z",
    "userTechnicalId": "1c1877ed-eb81-4f3d-893d-c2b6d8f43c38",
    "kWhCharged": 31,
    "chargingPlace": "Sorbonne_01"
  },
  {
    "date": "2025-07-11T06:09:41.332Z",
    "userTechnicalId": "1c1877ed-eb81-4f3d-893d-c2b6d8f43c38",
    "kWhCharged": 31,
    "chargingPlace": "Sorbonne_01"
  },
  {
    "date": "2025-07-11T06:09:41.332Z",
    "userTechnicalId": "1c1877ed-eb81-4f3d-893d-c2b6d8f43c38",
    "kWhCharged": 31,
    "chargingPlace": "Sorbonne_01"
  },
  {
    "date": "2025-07-11T06:09:41.332Z",
    "userTechnicalId": "f3946937-b6de-4631-9673-9b46e12855b0",
    "kWhCharged": 28,
    "chargingPlace": "Montmartre_02"
  },
  {
    "date": "2025-07-11T06:09:41.332Z",
    "userTechnicalId": "f3946937-b6de-4631-9673-9b46e12855b0",
    "kWhCharged": 28,
    "chargingPlace": "Montmartre_02"
  },
  {
    "date": "2025-07-11T06:09:41.332Z",
    "userTechnicalId": "9aa641a2-361e-4027-bd49-5c0c3de7f54e",
    "chargingPlace": "Sorbonne_02"
  },
  {
    "date": "2025-07-11T06:09:41.332Z",
    "userTechnicalId": "48053017-07b5-4d3b-ade4-e1183423453b",
    "kWhCharged": 52,
    "chargingPlace": "Grand_Hotel_Paris_01"
  },
  {
    "date": "2025-07-11T06:09:41.332Z",
    "userTechnicalId": "45f2389b-f8f6-4d1b-960a-19ab82b15b34---",
    "kWhCharged": 20,
    "chargingPlace": "Rue_Morgue_01"
  }
]

Output data

Below is a sample of output data in the Deduplicate function after it has applied the configured actions, which are to discard the duplicate records from every column in the input data.

Deduplicate function after it has discarded duplicated records from every column

[
  {
    "date": "2025-07-11T06:09:41.332Z",
    "userTechnicalId": "1c1877ed-eb81-4f3d-893d-c2b6d8f43c38",
    "kWhCharged": 31,
    "chargingPlace": "Sorbonne_01"
  },
  {
    "date": "2025-07-11T06:09:41.332Z",
    "userTechnicalId": "f3946937-b6de-4631-9673-9b46e12855b0",
    "kWhCharged": 28,
    "chargingPlace": "Montmartre_02"
  },
  {
    "date": "2025-07-11T06:09:41.332Z",
    "userTechnicalId": "9aa641a2-361e-4027-bd49-5c0c3de7f54e",
    "chargingPlace": "Sorbonne_02"
  },
  {
    "date": "2025-07-11T06:09:41.332Z",
    "userTechnicalId": "48053017-07b5-4d3b-ade4-e1183423453b",
    "kWhCharged": 52,
    "chargingPlace": "Grand_Hotel_Paris_01"
  },
  {
    "date": "2025-07-11T06:09:41.332Z",
    "userTechnicalId": "45f2389b-f8f6-4d1b-960a-19ab82b15b34---",
    "kWhCharged": 20,
    "chargingPlace": "Rue_Morgue_01"
  }
]