Data Veracity for Correcting Data

Data Veracity provides a repository of erroneous records and allows users to search, view, modify and reprocess the data. Data Veracity will serve as an alternative to the Error Correction System (ECS), being able to complement and improve upon the existing functionalities. Data Veracity will be able to support larger volumes of data and has a higher performance capability when it comes to searching and filtering compared to the ECS. However, unlike the ECS, Data Veracity will only support erroneous UDRs, as opposed to both erroneous Batch files and UDR.

Data Veracity will contain a new web UI to support data exploration, classification and modification of records. this is accompanied by 3 new agents and APL functions to offload data flows from invalid, corrupt or incomplete records. The APL functions will work much like the ECS functions to facilitate the automation of detecting, qualifying and correcting the invalid records.

Data Veracity Profile

The Data Veracity Profile contains settings for selecting the database profile, mapping of UDR types to be used, mapping information for MIM and generation of the SQL script for the generation of tables in the external database.

Data Veracity is supported for use with the following databases:

• Oracle
• PostgreSQL

The Data Veracity forwarding agent will be able to send any erroneous UDRs collected from any particular source and connect to the external database to insert and commit the UDRs and any related metadata into the specific table. A standard Data Veracity Forwarding workflow will typically involve a collection agent, an analysis or aggregation agent to perform the UDR filtering, followed by the Data Veracity forwarding agent itself to output the erroneous UDRs.

Example of Workflow with the Data Veracity forwarding agent

The Data Veracity collection agent will be used to collect any erroneous UDRs from the database using the search filters or error codes defined in the Data Veracity web UI. The UDRs to be collected will have either the NEW or UPDATED states in the database table, the state will be changed from NEW to UPDATED after collection. A standard Data Veracity collection workflow will typically involve the Data Veracity collection agent, an analysis agent or aggregation agent to reprocess the UDR, followed by a standard output agent or a Data Veracity forwarding agent to forward any remaining erroneous UDRs back to the database tables.

Data Veracity Web UI

The web UI for Data Veracity will contain 6 functions for the users to perform and these serve to allow the user to manipulate the erroneous data, manage all aspects of filtering, error codes and repair rules.

Data Veracity Web UI