Data Remediation
Data Remediation is where you apply rules or rule sets to improve data quality in a controlled and transparent way. This workflow is designed to help you validate changes before execution, confidently remediate data, and clearly understand the impact of each run.
Dry Run (Preview Remediation Impact)
Before making any changes to your data, you can perform a dry run. This allows you to simulate the execution of selected rules or rule sets without modifying the dataset.
During a dry run, you can:
See how many records would be affected by each rule
Review sample rows that match rule conditions
Validate logic and scope to avoid unintended changes
Execute Remediation
Once you are satisfied with the dry-run results, you can execute remediation to apply the rules to the dataset. The system processes the data based on the defined conditions and actions, ensuring changes are applied consistently and reproducibly.
Remediation Output
After executing a remediation run, Data Quality Manager generates a set of outputs to help you review, validate, and reuse your results. All outputs are automatically saved to your project in a structured directory for traceability and easy access.
Note: By default, the Remediation Output will be uploaded to the project where you launched the job. You can change these settings in a configuration file as instructed in the Remediation Configuration (Optional) section.
Remediated Data
This dataset contains the cleaned and transformed data after applying the selected rules. The output is generated as a new dataset and does not overwrite the original input, ensuring full data lineage and audit-ability.
Path: /Data_Quality_Manager_Output/YYYY-MM-DD/RUN_YYYY-MM-DD-Time-ID/Remediated_Data/<table>.csv
Data Quality Assessment Report
This report summarizes data quality metrics before and after remediation. It highlights improvements across dimensions such as completeness, consistency, and validity, and provides detailed insights into rule performance and impact.
Path: /Data_Quality_Manager_Output/YYYY-MM-DD/RUN_YYYY-MM-DD-Time-ID/report_ID.html
Dictionaries and Coding Data
A data dictionary, entity dictionary, and coding data are always generated for the remediated dataset. These outputs describe the dataset schema, column properties, and categorical mappings, supporting downstream analysis like DNAnexus Apollo data ingestion.
Paths:
Data Dictionary:
/Data_Quality_Manager_Output/YYYY-MM-DD/RUN_YYYY-MM-DD-Time-ID/DML/dx_data_dictionary.csvEntity Dictionary:
/Data_Quality_Manager_Output/YYYY-MM-DD/RUN_YYYY-MM-DD-Time-ID/DML/dx_entity_dictionary.csvCoding Data:
/Data_Quality_Manager_Output/YYYY-MM-DD/RUN_YYYY-MM-DD-Time-ID/DML/dx_codings.csv
Applied Rules JSON File
A JSON file that captures all rules applied during the remediation run, including their configurations and logic for transparency.
Path: /Data_Quality_Manager_Output/YYYY-MM-DD/RUN_YYYY-MM-DD-Time-ID/applied_rules.json
Remediation Configuration (Optional)
You can customize how remediation runs are executed by providing a configuration file. This allows you to control the below settings:
DATA_QUALITY_MANAGER_OUTPUT: Directory where all remediation outputs are saved.
DATA_QUALITY_MANAGER_REPORT: If specified, the Data Quality Assessment Reports are cloned and stored in this directory.
DATA_QUALITY_MANAGER_PROFILING_DATA: Directory where profiling data is stored.
DATA_QUALITY_MANAGER_SCHEMA_MAPPING: Directory containing target schemas (data dictionaries) used for source-to-target transformations.
DATA_QUALITY_MANAGER_SOURCE_SYSTEM: List of source system names for the input data.
DATA_QUALITY_MANAGER_TARGET_SYSTEM: List of target system names for the remediated output.
To do this, create a configuration file named .config and save it in the .Data_Quality_Manager folder of your project. The file should contain the following information:
Where project-xxxx is the ID of the project you want to export the output data to.
If no configuration file is provided, default settings (as in the Remediation Output section above) will be applied.
This approach enables more flexible and consistent execution, especially when running remediation workflows across multiple datasets or projects.
Last updated