> For the complete documentation index, see [llms.txt](https://academy.dnanexus.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://academy.dnanexus.com/data-quality-manager/common-use-cases.md).

# Common Use Cases

### PHI Detection and De-identification

This workflow allows users to identify Protected Health Information (PHI) within a dataset, review and correct the metadata, and establish reusable de-identification rules for future automated processing.

#### Step 1: Input and PHI Detection

1. Navigate to the Data Profiler from the Home Page.
2. Select your input Table data (individual files or an entire folder) and provide a Dataset name.
3. Optional: Include a Data dictionary, Entity dictionary, and Coding data to capture structural relationships and metadata.
4. Click Profile to launch the automated data assessment, which includes automatic PHI detection across all columns.

#### Step 2: Review and Corrections

1. Once profiling is complete, navigate to the Dataset Overview within the Data Profiler.
2. Review the Column Summary, which indicates the likelihood of Protected Health Information of each column.
3. Manually correct any false positives (data flagged as PHI that is safe) or false negatives (missed PHI) by modifying the data dictionary properties under Edit Schema.
4. Click Profile to apply your metadata corrections and generate an updated dataset version.

**Step 3: Define De-identification Rules**

1. Move to the Data Refinery tab to establish remediation logic.
2. Select Create New Rules.
3. Utilize predefined, custom rules tailored for specific PHI types to execute required obfuscation, masking, or removal actions.
4. Navigate to Rule Sets to save these rules into a cohesive Rule Set for repeatable use.
5. Go to Refinery to apply the created rule set and evaluate the PHI De-identification results.

**Step 4: Non-Interactive / Batch Automation**

Once a Rule Set is established for a specific data schema, you can automate future de-identification tasks on similar datasets via non-interactive batch execution using the DNAnexus CLI.

Example Command:

```
dx run app-data_quality_manager \
-itable_files="/data/remediation_test/patients.csv" \
-itable_files="/data/remediation_test/visits.csv" \
-itable_files="/data/remediation_test/results.csv" \
-idataset_name="New Clinical Batch" \
-irule_set_id="xxxxxxxxxxx" \
-y --name "Automated PHI Clean"
```

Required Batch Parameters:

* table\_files: Path (or ID) of the input table to include in the dataset
* dataset\_name: Name for the newly created target dataset.
* rule\_set\_id: The unique identifier of your saved PHI De-identification Rule Set.

### Apollo Phenotypic Data Ingestion

This workflow allows users to prepare raw tabular data for ingestion into an Apollo dataset using the DNAnexus [Data Model Loader](https://documentation.dnanexus.com/developer/ingesting-data/data-model-loader) (DML). The application automatically infers the required metadata schema, validates it, and triggers a seamless ingestion pipeline.

#### Step 1: Input and Metadata Inference

1. In the Home Page, provide a Dataset name and select your raw Table data (CSV, TSV, or Parquet files).
2. Click Profile. The application will automatically inspect your dataset and infer the metadata required by the Data Model Loader, generating a Data Dictionary, an Entity Dictionary, and a Coding File.
3. Review the data quality and summary at the Dataset Overview page.

#### Step 2: Review and Edit Schema

1. Navigate to Data Profiler → Edit Schema to verify the inferred data dictionaries, coding values, and table relationships.
2. Adjust properties such as column types, descriptions, and structural entity lookups directly in the metadata editor if needed.
3. Click Profile to apply the changes
4. Click Export to Apollo to verify the schema and datasets and launch a Data Model Loader (DML) job to ingest the data.

#### Step 3: Data Remediation (If Validation Fails)

1. Review the validation results.
2. If structural data inconsistencies, missing values, or relational errors violate DML constraints, switch to the Data Refinery tab.
3. Implement Transformation Rules (e.g., joins, column renaming, schema alignment) or SQL/Python Rules to standardize the raw data against the Apollo constraints.
4. Run a Dry Run to check your modifications, then Execute & Profile to generate clean output tables which are ready for Apollo ingestion.

#### Step 4: Launch DML Ingestion to Apollo

1. Once the schema is correct and all validation issues are resolved, the dataset is ready for ingestion.
2. Click Export to Apollo to verify the schema and datasets again and launch a DML job to ingest the data.
3. The Data Quality Manager will automatically bundle the finalized outputs—including any remediated data tables along with the generated dictionaries and coding files—and directly trigger a Data Model Loader (DML) job.
4. The successful output of this execution run is a fully ingested Apollo dataset available for downstream cohort browser analysis.

#### Step 5: Automation via Non-Interactive Mode

For future datasets that utilize an identical data schema, the entire remediation and Apollo ingestion pipeline can be completely automated using a single batch execution command through the DNAnexus CLI.

Example Command

```
dx run app-data_quality_manager \
-itable_files="/data/table1.csv" \
-itable_files="/data/table2.csv" \
-itable_files="/data/table3.csv" \
-idataset_name="Apollo Production Ingest" \
-irule_set_id="xxxxxxxxx" \
--ireport_type=dml \
-y --name "Automated Apollo Ingestion"
```

Key Ingestion Parameters:

* table\_files: File path(s) to the new raw tables matching the verified schema.
* dataset\_name: Name for the generated dataset created during the workflow.
* rule\_set\_id: The ID of your saved Refinery Rule Set that formats the data for DML.
* report\_type: Set to dml to trigger the Apollo database loading task.