Validate configuration
Validate is a processor function that operates on data as it passes through a stream. Rules can be configured on data fields in the user interface displayed in the Rules tab or written in JSON file format in the Schema tab. Most rules can be set up in the Rules tab, which automatically generates a JSON schema. Rules that cannot be created in the user interface can be defined in the Schema tab.
Note!
Rules configured in the Schema tab will be reflected in the Rules tab and vice versa. It is possible to configure
Invalid data handling
In this section, you choose how to handle records that fail validation. These options are available whether you configure rules in the Rules tab or the Schema tab. You can configure one of the following actions for invalid records:
Discard
Discarding records means they will be permanently removed and not sent elsewhere.
Note!
Discarding is mutually exclusive with the other options below.
Process invalid data - you can choose to:
Send the records toData correction - the default option.
Route the records to a new output for investigation.
Send the records to both Data correction and a new output.
A record is sent to Data Correction or the specified output as soon as the first validation rule fails. No further validation checks are performed after that.
Setting | Description |
|---|---|
Discard | Select to remove the invalid data from the stream |
Process invalid data | Select this option if you want to collect and manage the invalid data. Choose one or both of the following options:
 |
General validation rules
The Validate function uses normalization rules during validation, see Validate normalization rules. The following validation rules apply:
The Validate function does not perform strict validation on the input data types.
The record that passes through the validation function might be converted to the data type defined in the JSON schema.
Validation is done when the input data type can be converted.
Rules tab
In the rules tab Validation Rules section, you define validation rules on fields in the UI. Click + Add rule to create a new rule.
Setting | Description |
|---|---|
Field | |
Field name | Select the field from the drop-down list or add a field. |
Field type | Select the field type from the following options:
|
Description | Add a description for the validation rule (optional). |
This field is required | Check this box to mark the field as mandatory. If checked, the field must be present on the record; otherwise, validation will fail. |
Validations | |
Key | This is the rule that will be applied to the selected field. |
Value | This is the value for the key. The options will vary based on the selected key. |
Hint!
When configuring validation for nested fields, always define the most specific (deepest) fields first, followed by their parent objects.
For example, if you want to validate customer.info.postalCode, customer.info, and customer.address:
Start with a rule for
customer.info.postalCode(e.g., check that it's a valid postal code format)Then add a rule for
customer.info(e.g., ensure it's an object)Finally, validate
customer.address(e.g., ensure it's not null or matches a certain structure)
This ensures that deeper validations are not skipped due to missing or invalid parent fields.
Schema tab
The Schema tab allows you to define validation rules directly in JavaScript. This is useful for advanced scenarios or for creating rules that are not supported in the Rules tab.
Sometimes, rules defined in the schema may not be compatible with the visual interface in the Rules tab. If a mismatch occurs, an error message will appear when switching back to the Rules tab. The validation will still function as expected in the stream.
Caution!
When adding rules directly to the schema, you must click Apply rules for them to take effect.
Note!
Validation rules created in either the Rules or Schema tab are used in the same way by the system. Likewise, Invalid Data Handling settings apply regardless of which method you use to define your validation rules.