> For the complete documentation index, see [llms.txt](https://academy.dnanexus.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://academy.dnanexus.com/data-quality-manager/data-remediation.md).

# Data Remediation

Data Remediation is where you apply rules or rule sets to improve data quality in a controlled and transparent way. This workflow is designed to help you validate changes before execution, confidently remediate data, and clearly understand the impact of each run.

#### Dry Run (Preview Remediation Impact)

Before making any changes to your data, you can perform a dry run. This allows you to simulate the execution of selected rules or rule sets without modifying the dataset.

During a dry run, you can:

* See how many records would be affected by each rule
* Review sample rows that match rule conditions
* Validate logic and scope to avoid unintended changes

<img src="https://1979569080-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPtCOm9rXoRi4P9rh1ET8%2Fuploads%2F1uR6NrBd0ILumlT0Xl9s%2Funknown.png?alt=media&amp;token=b449812d-0a12-478b-91c6-a7bc61181aa3" alt="" height="300" width="576">

#### Execute Remediation

Once you are satisfied with the dry-run results, you can execute remediation to apply the rules to the dataset. The system processes the data based on the defined conditions and actions, ensuring changes are applied consistently and reproducibly.

### Remediation Output

After executing a remediation run, Data Quality Manager generates a set of outputs to help you review, validate, and reuse your results. All outputs are automatically saved to your project in a structured directory for traceability and easy access.

Note: By default, the Remediation Output will be uploaded to the project where you launched the job. You can change these settings in a configuration file as instructed in the **Remediation Configuration (Optional)** section.

#### Remediated Data

This dataset contains the cleaned and transformed data after applying the selected rules. The output is generated as a new dataset and does not overwrite the original input, ensuring full data lineage and audit-ability.

Path: `/Data_Quality_Manager_Output/YYYY-MM-DD/RUN_YYYY-MM-DD-Time-ID/Remediated_Data/<table>.csv`

#### Data Quality Assessment Report

This report summarizes data quality metrics before and after remediation. It highlights improvements across dimensions such as completeness, consistency, and validity, and provides detailed insights into rule performance and impact.

Path: `/Data_Quality_Manager_Output/YYYY-MM-DD/RUN_YYYY-MM-DD-Time-ID/report_ID.html`

#### Dictionaries and Coding Data

A data dictionary, entity dictionary, and coding data are always generated for the remediated dataset. These outputs describe the dataset schema, column properties, and categorical mappings, supporting downstream analysis like DNAnexus Apollo data ingestion.

Paths:

* Data Dictionary: `/Data_Quality_Manager_Output/YYYY-MM-DD/RUN_YYYY-MM-DD-Time-ID/DML/dx_data_dictionary.csv`
* Entity Dictionary: `/Data_Quality_Manager_Output/YYYY-MM-DD/RUN_YYYY-MM-DD-Time-ID/DML/dx_entity_dictionary.csv`
* Coding Data: `/Data_Quality_Manager_Output/YYYY-MM-DD/RUN_YYYY-MM-DD-Time-ID/DML/dx_codings.csv`

#### Applied Rules JSON File

A JSON file that captures all rules applied during the remediation run, including their configurations and logic for transparency.

Path: `/Data_Quality_Manager_Output/YYYY-MM-DD/RUN_YYYY-MM-DD-Time-ID/applied_rules.json`

### Remediation Configuration (Optional)

You can customize how remediation runs are executed by providing a configuration file. This allows you to control the below settings:

* **DATA\_QUALITY\_MANAGER\_OUTPUT**: Directory where all remediation outputs are saved.
* **DATA\_QUALITY\_MANAGER\_REPORT**: If specified, the Data Quality Assessment Reports are cloned and stored in this directory.
* **DATA\_QUALITY\_MANAGER\_PROFILING\_DATA**: Directory where profiling data is stored.
* **DATA\_QUALITY\_MANAGER\_SCHEMA\_MAPPING**: Directory containing target schemas (data dictionaries) used for source-to-target transformations.
* **DATA\_QUALITY\_MANAGER\_SOURCE\_SYSTEM**: List of source system names for the input data.
* **DATA\_QUALITY\_MANAGER\_TARGET\_SYSTEM**: List of target system names for the remediated output.

To do this, create a configuration file named `.config` and save it in the `.Data_Quality_Manager` folder of your project. The file should contain the following information:

```
DATA_QUALITY_MANAGER_OUTPUT = project-xxxx:/Data_Quality_Manager_Output 
DATA_QUALITY_MANAGER_REPORT = project-xxxx:/Data_Quality_Manager_Report
DATA_QUALITY_MANAGER_PROFILING_DATA = project-xxxx:/Profiling_Data
DATA_QUALITY_MANAGER_SCHEMA_MAPPING = project-xxxx:/.Schemas_Mapping
DATA_QUALITY_MANAGER_SOURCE_SYSTEM = SOURCE_1, SOURCE_2
DATA_QUALITY_MANAGER_TARGET_SYSTEM = TARGET_1, TARGET_2
```

Where project-xxxx is the ID of the project you want to export the output data to.

If no configuration file is provided, default settings (as in the Remediation Output section above) will be applied.

This approach enables more flexible and consistent execution, especially when running remediation workflows across multiple datasets or projects.