The Archiving agent provides comprehensive data management capabilities. This agent, profile and inspector support both archival and retrieval processes.

• Looking up Archived Data: Customers can use the Archive Inspector to look up archived files. This tool is available in the Execution Manager and provides advanced search functionality that allows you to search for files within specified periods.

• Deleting Archived Data: The Archive Cleaner task is responsible for automatic deletion of outdated archives based on the purge criteria set within the Archive profile. You can also trigger clean-up processes manually to ensure that the data does not exceed its intended retention period.

For more information on functionality and configuration, see the Archiving section in the MediationZone software description.

The Amazon S3 forwarding agent provides capabilities to write data to S3.

For best practices around data security and encryption, see the Amazon S3 documentation.

• Looking up Records:

  • Data Veracity User Interface: Open the Data Veracity user interface to search and locate specific UDRs or batches. Use the Search options to filter records based on various criteria such as error codes, insert times, or specific MIM values. With advanced filtering and saved filters, you can streamline recurrent searches and improve efficiency in managing records.

  • View and Inspect Data: Once records are located, you can view details and inspect the contents of UDRs in the user interface.

• Deleting Records:

  • Mark for Deletion: If a record contains sensitive information that should not be retained, mark it for deletion directly in the Data Veracity user interface. This action flags the records but does not remove them from the database.

  • Force Deletion: For immediate removal, use the Force Delete option available to administrators. This option should be used with caution, as it permanently removes the record from the database, ensuring that sensitive data is irrecoverable.

  • Automated Cleanup via Maintenance Task: Configure the DataVeracity_Maintenance system task to automatically remove records that have been marked for deletion. This task can be triggered based on specific conditions and time frames, adhering to data retention policies.

• Managing Access and Security:

  • Restricted Fields and Data Masking: Ensure sensitive fields are restricted and masked to prevent unauthorized access and modification. With Data Veracity you can configure permissions to restrict who can view or edit sensitive data.

  • Permission Controls: Use the Access Controller to assign and manage permissions for different user levels, ensuring that only authorized personnel can execute sensitive operations like data deletion or modification.

• Auditing and Compliance:

  • Audit Trails: Keep track of who accessed and modified the Data Veracity records. Ensure all actions on sensitive data are logged to support compliance audits and internal controls.

• External Databases:

See the specific documentation provided by each database platform provider. Implement encryption and secure data handling practices as recommended:

• PostgreSQL: PostgreSQL Documentation

• Oracle Database: Oracle Security Guides

• SAP HANA: https://help.sap.com/docs/

For more information on functionality and configuration, see the Data Veracity section in the MediationZone documentation.

• Looking Up Records:

  • Search for Records: Use the ECS Inspector's Search function to find specific UDRs or batches. You can filter searches based on various criteria such as workflow, agent, error code, or insert period. Advanced search options allow for more detailed filtering, including the use of wildcards and intervals for specified fields.

  • Using Saved Filters: For frequent searches, you can save your search settings as filters for quick future access. This is particularly useful for regularly monitoring or auditing records that meet specific criteria.

• Deleting Records:

  • Selective Deletion: After locating the records using the Search function, you can selectively delete entries. Ensure that the records are in a reprocessed state, as only reprocessed entries can be safely removed without affecting ongoing processes.

  • Bulk Deletion: For managing larger volumes of data, the Bulk Edit feature allows you to delete multiple records simultaneously. This tool is ideal for removing batches of sensitive information efficiently.

  • Scheduled Cleanup: Use the ECS Maintenance System Task to set up automatic purging of outdated or unnecessary records. This can be configured to remove records based on their reprocessing state and can be tailored to ensure compliance with data retention policies.

• Secure Management:

  • Restrict Field Edits: To prevent unauthorized modifications, configure restricted fields to protect sensitive data within UDRs from being edited.

  • Monitor and Audit: Regularly review the ECS Maintenance logs and use the statistics reports to ensure that deletions and data management practices comply with organisational security policies.

For more information on functionality, see th Error Correction System section in the MediationZone documentation.

• Using SQL Processing Agent for Lookup Operations:

  • Query Configuration for Data Retrieval: Configure the SQL Processing Agent to perform selective queries that identify sensitive data needing review or deletion. This involves setting up SQL queries within the agent to filter and retrieve only the data matching specific security or compliance criteria.

• Configuring the SQL Forwarding Agent for Data Deletion:

  • SQL Statement Configuration: Use the SQL Forwarding Agent to execute SQL DELETE commands. In the agent’s configuration dialog, enter SQL statements specifically designed to target and delete sensitive records. Example: DELETE FROM customer_records WHERE customer_id = $(UDR.CustomerID);.

  • Dynamic Data Handling: Leverage MIM values and UDR field variables in your SQL statements to ensure that only the intended records are targeted for deletion, enhancing security and precision.

  • Stored Procedures Support: For complex deletion requirements, configure the agent to call stored procedures that encapsulate deletion logic. This approach enhances security by abstracting the deletion logic into the database layer. Example SQL call: CALL secure_delete_procedure($(UDR.CustomerID));.

For more information on functionality and configuration, see the SQL Agents section in the MediationZone documentation..

• Secure Configuration:

  • Directory Security: Configure the Disk forwarding agent to store files in secured directories. Ensure that these directories have appropriate permissions set to prevent unauthorized access. Use secure file system permissions and regularly audit access rights.

  • Path Configuration: Carefully define and regularly review the paths used for storing files both temporarily and permanently. Avoid common directories and ensure paths are not publicly accessible.

• File Handling Procedures:

  • File Creation and Management: When the Disk forwarding agent receives a 'Begin Batch' message, it should create files in a designated temporary directory, moving them to a permanent location only once fully processed.

• Data Minimization and Retention:

  • Minimize Data Exposure: Configure the agent to produce files only when necessary. Avoid storing sensitive information unless required. Use the Produce Empty Files option to prevent the creation of unnecessary data files.

  • Retention Policy: Define and enforce a data retention policy specifying how long files should be retained in both temporary and permanent storage. Automate the deletion of files that are beyond their retention period to prevent accumulation of outdated sensitive data.

• Command Execution Security:

  • Command Configuration: If commands are configured to run after file closure, ensure these commands do not expose sensitive data or interact with unsecured external systems. Validate and sanitize all command inputs to prevent execution vulnerabilities.

• Error Handling and Incident Response:

  • Error Handling: Configure the agent to handle errors without data leakage. For instance, if a Cancel Batch message is received, ensure that all associated data is securely deleted without leaving any traces in the temporary directory.

For more information on functionality and configuration, see the Disk Forwarding Agent - Batch section in the MediationZone documentation.

• Secure Configuration and Access Controls:

  • Database Profile Management: Ensure that the database profiles are securely configured and reviewed. Use encrypted connections to the database to protect data in transit. Access to the database should be restricted to authorized users and systems only.

  • Default Schema Use: Limit the use of default schemas in databases, especially for operations involving sensitive data. Specify explicit schemas that segregate data access based on user roles and data sensitivity.

• Data Handling and Integrity:

  • Direct Insertions vs Stored Procedures: Use stored procedures instead of direct insertions whenevr possible, as they allow for better control of the data manipulation logic and can encapsulate business rules and data validation steps.

  • Transaction ID Management: Implement transaction ID management to ensure that each batch operation can be uniquely identified and managed. This helps in maintaining data integrity, particularly in rollback scenarios and inter-workflow communications.

• Data Minimization and Retention:

  • Field Mapping and Data Exposure: Minimize data exposure by configuring the agent to only handle necessary data fields. Sensitive data should be handled with extra care, potentially encrypting data before insertion.

  • Data Retention Policies: Define data retention policies that specify how long data should be retained within the database. Automate the cleanup of old data to comply with these policies, ensuring that data is not kept longer than necessary.

• Error Handling and Cleanup Procedures:

  • Error Handling: Configure the agent to handle SQL exceptions and errors. Ensure that operations such as Run SP (Stored Procedure) at the end of data forwarding incorporate transaction safety measures to prevent data leakage or corruption.

  • Cleanup Operations: Use Cleanup SP to define stored procedures that clean up data in case of workflow cancellation or error. This prevents orphaned data from accumulating.

• Audit and Monitoring:

  • Transaction Auditing: Enable detailed auditing for all transactions processed by the Database forwarding agent. Audit logs should capture key details about the transactions, including transaction IDs, timestamps, and user IDs wherever applicable.

For more information on functionality and configuration, see the Database Forwarding Agent section in the MediationZone documentation.

• Data Handling and Security:

  • Secure Data Transfer: Ensure all data transfers are conducted over secure channels. For SFTP and SCP, use secure protocols like SSH2 for encryption.

  • Cryptography: Regularly update and manage cryptographic measures such as public and private keys, ensuring that keys are rotated and managed according to security policies.

• File Management and Retention:

  • Post-Transfer Policies: Define clear policies for the handling of files post-transfer, including moving, renaming, or deleting files as necessary. This should include criteria for automatic deletion based on retention schedules.

  • Post-Processing File Management: Use the functionalities of agents to automatically manage files after processing, such as decompression or removal, to minimize data exposure.

• Error Handling and Retry Mechanism:

  • Error Handling: Implement error handling to manage and log transfer failures or data processing issues. Use the agent's capabilities to retry connections and resume interrupted transfers securely.

  • Prevention of Data Corruption/Loss: Configure agents to handle decoding errors and connection retries appropriately to prevent data corruption or loss.

• Access and Authentication Controls:

  • Authentication: Ensure that all access to data transfer tools and interfaces is controlled via authentication mechanisms like passwords or SSH keys. Configure agents to require authentication for both initiating transfers and accessing the data.

  • Prevention of Unauthorized Access: Limit retries and re-exchanges to prevent unauthorized access attempts and ensure that security settings like host key verifications are enforced.

• Audit and Compliance:

  • Auditing Data Transfer: Regularly audit data transfers, access logs, and security settings to ensure compliance with data protection policies. Use built-in logging and event management in agents to track and monitor all activities.

For more information on functionality and configuration, see the Batch-Based Real-Time Agents section in the MediationZone documentation.