Search & Repair(3.3)

Owned by Yuri Au Yong
Last updated: Oct 04, 2023
10 min read

The Search & Repair UI allows users to perform the following:

  • Search for Data Veracity records (erroneous UDRs or batch files)

  • Change the state

  • Mark Data Veracity records for deletion

  • Force delete Data Veracity records

  • Repair Data Veracity records

Data Veracity - Search

Searching for UDR Input Type

To perform a search for UDR Input Type in Data Veracity, follow the steps below:

Unsupported Data Type for Search

  • Searching or filtering of UDRs with list and map data types are currently not supported by Data Veracity.

  • Searching by filter with Bit Set Field data type is not supported by Data Veracity as well.

  1. To search for UDR Input Type, you must first select a Data Veracity profile. Click the Filter button to open the Search dialog. 


    Data Veracity Search dialog

  2. Click Browse to select the Data Veracity profile.

  3. Select UDR for Input type.

  4. When you have selected UDR for Input type, you can configure the following:

    1. UDR types - Select the specific UDR from a list. You can only choose ONE UDR type to access.
       


      Data Veracity UDR types combo box

    2. Display Fields - Select specific fields (which includes meta data, user defined MIMs and UDR fields) that you would like to display in the query results.


      Display Fields selection dialog box

    3. Search Query - Add the filter rules that you want in the query builder by clicking on Add. If more than one rule is defined, you would be able to choose between AND or OR by clicking on either of the buttons. You could also add a new group of rules by clicking on the Add Group button to define a nested set of rules for the query. When you run a query, rows in the table that matches the expression will be displayed in the Query Results. 
      For more information, see the Search Query section below.

  5. Click OK to execute the query and the results will be displayed.


    Example Query Result for UDR Input type

Query Results Sort Order

Sorting of search results is done from the Data Veracity Database via SQL query statement and so the sorting behavior will depend on the behavior of the Data Veracity Database(for example, sorting of alphabetical characters in Oracle is case sensitive, whereas in PostgreSQL it is case insensitive).

Data Veracity State column values is represented as numbers in the Data Veracity Database. State column will be sorted by the order of the numbers assigned to the states where:

  1. NEW

  2. UPDATED

  3. PRE_DELETE

  4. REPROCESSED

  5. DELETE_APPROVED

Searching for Batch Input Type

To perform a search for Batch Input Type in Data Veracity, follow the steps below:

  1. To search for Batch, you must first select a Data Veracity profile. Click the Filter button to open the Search dialog. 


    Data Veracity Search dialog

  2. Click Browse to select the Data Veracity profile.

  3. Select Batch for Input type.

  4. When you have selected Batch for Input type, you can configure the following:

    1. Display Fields - Select specific fields (which includes meta data and user defined MIMs) that you would like to display in the query results.


      Display Fields selection dialog box

    2. Search Query - In the Search Query panel, add the Filter rules that you want in the query builder by clicking on Add. If more than one rule is defined, you would be able to choose between AND or OR by clicking on either of the buttons. You could also add a new group of rules by clicking on the Add Group button to define a nested set of rules for the query. For more information, see the Search Query section below.

  5. Click OK to execute the query and the results will be displayed.


    Example Query Result for Batch Input type

Query Results Sort Order

Sorting of search results is done from the Data Veracity Database via SQL query statement and so the sorting behavior will depend on the behavior of the Data Veracity Database(for example, sorting of alphabetical characters in Oracle is case sensitive, whereas in PostgreSQL it is case insensitive).

Searching with Saved Filters

To perform a search by loading a previously saved filter, follow the steps below:

  1. To search with a saved filter, you must select a previously saved filter. Click the Filter button to open the Search dialog. 

  2. Search and select a saved filter with the Saved filters drop down and the configured filter values will all be loaded in the Search dialog.

  3. Click OK to execute the query and the results will be displayed.

Search Query

Search Query builder

This panel represents an expression based on the columns in the Data Veracity table. When you execute a query, rows in the Data Veracity table that matches the expression will be displayed. 

Item

Description

Item

Description

AND

Click this button to apply AND logic on a set of rules or groups.

OR

Click this button to apply OR logic on a set of rules or groups.

Field name

This drop-down list contains columns in the selected database table that you can use in your query.

The query is carried out based on this selection as a filter rule to populate the result for the Display Field.

Example field names

Operator

This is a comparison operator that is applied to the values of the specified column.

The available operators are listed according to the data type of the selected column.

Example operators drop-down

Value

This field contains a value that is used for comparison with the selected column. When between and not between is the selected operator, two value fields are displayed.

When you click this field and the column is of a date type, a date picker will be displayed.

Add

Click this button to add a new rule to the expression.

Add Group

Click this button to add a group of rules within the expression. The groups can be nested.

Remove Group

This button will only appear when a group is added.

Click this button to delete a group. When you delete a group, the rules contained within it will be deleted as well.

Delete

Click this button to delete a rule.

Additional Actions

Apart from searching for Data Veracity records in the Data Veracity tables, you can perform the following actions in the Search & Repair. These actions buttons/menus can be found on the query results table. 

Marking Records for Deletion

UDRs or Batch files could be marked for deletion using the Mark Delete option. Executing this option will give the selected records the PRE_DELETE state and make it available in the Approve Delete UI. For more information about how deletion is approved, , refer to Approve Delete.

To mark records for deletion, you can

  • click on Mark Delete at the Actions column to mark a single record for deletion.

  • select the records you will want to mark for deletion then click on Mark Delete button in the result table menu bar.

  • click on Mark Delete button in the result table menu bar when all of the records are deselected if you wish to mark for deletion all the records in the result table.

A message will be displayed asking if you are sure that you would want to mark the selected records for deletion. If you click OK, the selected records will have their states marked as PRE_DELETE.

Repairing UDRs

This is only applicable when Input type is UDR. UDRs stored in Data Veracity could be edited and updated by the user to rectify any erroneous values found in the query results table. The repair job is conducted following these steps:

  1. To repair UDRs values, you can

    • Select the Data Veracity records of UDRs in which the user would like to repair the values, then click on the Repair button in the result table menu bar.

    • click on Repair button(from the 3 dots menu) in the result table menu bar when all of the records are deselected if you wish to repair all the UDRs in the result table.


  2. The New Repair Rules dialog box will appear. From this dialog box, you can create and save a new rule. Alternatively, you can load an existing rule to use or even to modify it. There can be one or multiple rules for each repair rule in Data Veracity. For more information about managing Repair Rules refer to Repair Rules.

    Repair rules dialog box

  3. To create a new rule, click on the New Rule button to begin:

    • You will select the UDR Field in the first combo box.

    • You will then select the operation to be conducted in the second combo box.

    • You can also check the APL check box to allow the include business logic in the form of APL codes into the rule itself.



  4. To remove a rule, click on the Minus button on the right side of each rule.

  5. Clicking on the Save button will save the Repair Rule as a new rule, but if the Repair Rule is already an existing rule, you have the option to save it as a new repair rule or to update the existing rule.

  6. Click the Repair Selected or Repair All button to execute the repair job. It will also take you to the Repair Jobs page where the repair job will be listed and the user will be able to monitor the progress of the job. For more information on the Repair Jobs page, please refer to Repair Jobs.

View UDRs

This is only applicable when Input type is UDR. Click on View UDR at the Actions column to inspect a UDR and view the field and values present in the UDR.

Unknown Attachment
Example - Viewing a UDR

Delete Records

Delete is an operation for the user to completely remove the Data Veracity record from the Data Veracity database tables without marking the record for deletion.

To delete records, you can

  • click on Delete at the Actions column to delete a single record.

  • select the records you will want to delete then click on Delete button in the result table menu bar.

  • click on Delete button(from the 3 dots menu) in the result table menu bar. when all of the records are deselected if you wish to delete all the records in the result table.

A message will be displayed asking if you would want to permanently remove the selected records. If you click OK, the selected records will be completely removed from the Data Veracity database tables.

Setting the State of a Data Veracity Record

Set state is an operation for the user to manipulate the state of the records. There are a total of 5 different states in Data Veracity: NEW, UPDATED, PRE_DELETE, REPROCESSED and DELETE_APPROVED. Each of these states signify the current status of the records where:

  • NEW is for newly inserted records from the Data Veracity Forwarding Agent or from cancelled batches.

  • UPDATED is for records that has been through a successful repair job execution.

  • PRE_DELETE is for records that have been marked for deletion.

  • REPROCESSED is for records that have been collected by the Data Veracity Collection Agent.

  • DELETE_APPROVED is for records that have been approved for deletion by an administrator.

To set a state for records, you can

  • select the records you will want to delete then click on Set State button in the result table menu bar, then choose from the five states to set the records to.

  • click on Set State button(from the 3 dots menu) in the result table menu bar when all of the records are deselected if you wish to set state for all the records in the result table, then choose from the five states to set the records to.

A message will be displayed asking if you are sure that you want to change the state to the selected new state. If you click OK, the selected records will have their states changed.

Export Data Veracity Records

Data Veracity allows the user to export all records in the search result into a zipped file.

  1. Click the Export button in the result table menu bar.

  2. In the Export search result dialog, enter the following:

    • Filename - The export filename should contains alphanumeric- or dash (-) characters. Do not add a file extension or a path.

    • Delimiter - You can enter a character or string that should represent the separator token, e.g. a comma or a semi-colon. 


      Export search result dialog
       

  3. Select the check box Include Header to add the column names to first row in the file.
     

  4. Click the OK button. A dialog box containing a link to download the zipped file will be displayed. All the columns of the table that is specified in Display Field will be included in the file.