/
Deduplicate configuration

Deduplicate configuration

CE.png

To configure the function, either double-click the function or select the function and click the Edit button.

Settings

DeduplicateSettings.png
Deduplicate processor configuration

You can configure this function with the following settings:

Setting

Description

Setting

Description

General

Checking options

  • Every column - Select to check for duplicates in all columns.

  • Specific columns - Select specific columns from the drop-down list in Check these columns, or type the fields one by one into the Add field box and click on the + button.

    SpecificColumns.png
    Checking options configuration

Records removed from cache after

Records are discarded from the cache after the number of days that you have specified in Records removed from the cache. Specify the number of days after which the records must be removed from the cache memory. The default value is one day. Records in the cache memory can be stored in the database for up to 70 days.

Caution!
This number should be kept as low as possible for performance reasons.

Handling duplicates

  • Discard - Select to discard the duplicates.

  • Create new output - Select to add a second output channel for duplicates. This lets you examine the duplicates and act upon them if required.

Note!

When you modify the settings, the records that are already stored in the cache are not affected. Your changes to the settings only apply to records that are processed after you have changed the settings.