Validate

The Validate Function allows you to configure validation rules to check for data accuracy of single or multiple fields in the data. It is important to detect errors before data is processed further, thus it is recommended to add the Validate Function to your stream where the data enters Cloud Edition.

Sample stream including the Validate function

The validation process works by adding a JSON schema to determine if a record is valid or not. The JSON schema describes the structure or content or both of a record and discards records that do not match the schema. Validation fields are searchable.

There are two ways of interacting with this node – a Rules selection using a dropdown menu entry or by placing code in the Schema section.

Configuration

There are two subsections available in the JSON Rules selection: Fields and Objects.

There is a distinction between the JSON objects and fields according to the format specifications.

Objects are data encapsulated in the JSON string format that contain key/value pairs of information. The keys themselves must be of the “string” data type and the values must correspond to a valid JSON data type.
Fields denote the name/value pairs that are placed in the objects.

Using the relevant “Add Property” or “Add Object” the desired items can be added to the service.

Field Rules Selection

When the “+ Add field” button is clicked, users can select fields from a drop-down list or type the fields one by one into the Add field box and click the + button. A description can be assigned to each property as well.

The user interface allows customers to choose between adding in the Output of the previous function or manually specify a field property of their choice by toggling the desired option.

Validation rules can be entered by clicking on the “Add Validation Rule” button. This adds a row, comprising a Key dropdown menu choice with an assigned Value input field box. Additional lines can be created by pressing the “Add Validation Rule” button again while existing rules can be removed by clicking on the remove icon on the selected row.

Data Correction

The records that fail validation are sent to Data Correction for further investigation. A record enters data correction when the first validation rule fails and the subsequent validation checks are not done. To enable data correction, select the Send invalid data to Data Correction checkbox.

Validation Types

The Validate Function can process instances of different JSON schema types. The current JSON schema implementation supports all data types (number, integer, string, boolean, array, object, or null) by the official specifications.

The Validate Function is interactive, if incorrect values are entered, an appropriate error will be shown.

Validation Rules

The Validate Function uses Normalization Rules during validation. The following validation rules apply:

The Validate Function does not perform strict validation on the input data types.
The record that passes through the validation function might be converted to the data type defined in the JSON schema.
Validation is done when the input data type can be converted.