Getting Started with Data Quality Manager
A license is required to access the Data Quality Manager on the DNAnexus Platform. For more information, please contact DNAnexus Sales (via sales@dnanexus.com).
Accessing Data Quality Manager on DNAnexus Platform
There are two ways to run the application:
Direct Access: Go to this link to open the app and launch a job.
Platform Navigation: Click on the top navigation bar, then select Open Tools > search for the Data Quality Manager app, and run the app.
The app requires no input to launch a job, input files can be specified in the job once it’s ready.
Open the Data Quality Manager UI
After launching a job, you will be redirected to the Monitor screen. From there, click the job name to view the job details.
It may take a few minutes for the job to be ready. The Data Quality Manager is an HTTPS application on the DNAnexus Platform, so it is accessed via the Job URL. Once the job is ready, click Open Worker URL to open the job UI.
If you attempt to click the button before the job is ready, you may encounter a "502 Bad Gateway" error. This is not a problem, it simply means you need to wait a bit longer before the environment is fully prepared.
Access the Home Page
After clicking the Open URL button to open the Data Quality Manager job, you will first arrive at the Home Page, which serves as the central entry point to all core components of the application.
Data Quality Manager homepage
From the Home Page, you can access:
Data Profiler to inspect data structure, review quality metrics, and identify data issues of all datasets that have been profiled in the project.
Data Refinery to define remediation rules and apply them to your data.
Data Monitor to review data quality trends and remediation history.
High-Level Recommended Workflow
Data Quality Manager high-level workflow
Input Specification
To use Data Profiler, you need to provide a dataset name and select the required Table data. You can also optionally include a Data dictionary, Entity dictionary, and Coding data (similar to those needed for Apollo data ingestion with the Data Model Loader). See the table below for more details.
You can provide input data by selecting either individual files or an entire folder
Input name
Required/
Optional
Input format supported
Description
Dataset name
Required
N/A
Name of your dataset.
Table data
Required
CSV files
TSV files
Parquet files
This is the data that will be profiled by this application. Each file is a table in your dataset.
You can provide input data by selecting individual files or an entire folder. When selecting a folder, all supported files within that folder are included in the dataset.
Data dictionary
Optional
A CSV file
The Data Dictionary file of the dataset. This file follows the format of DNAnexus Data Model Loader dictionary (Details).
If not provided, the relationship between tables in the dataset will be inferred.
Entity dictionary
Optional
A CSV file
The dictionary describes tables/entities that the data files represent (Details).
Coding data
Optional
A CSV file
This file must contain the coding_name header to link categorical fields. These names must match the corresponding values in the Data Dictionary (Details).
Working with Multiple Datasets
You can work with multiple datasets within a single job, or access datasets that were profiled in previous jobs within the same project.
Each dataset is processed independently, allowing you to:
Profile data quality for each dataset separately
Apply different remediation rules or rule sets per dataset
Manage and review datasets across runs in a consistent workflow
You can profile multiple datasets and access profile of other
Advanced Usage: Non-Interactive / Batch Execution
Data Quality Manager also supports a non-interactive execution mode through the DNAnexus CLI. This allows you to launch remediation jobs programmatically without accessing the interactive UI, making it suitable for batch processing, workflow automation, and large-scale operational runs.
In non-interactive mode, you can:
Specify input datasets directly from the command line
Apply an existing Rule Set automatically
Launch multiple remediation jobs in batch workflows
Integrate Data Quality Manager into automated pipelines
Example Command
Parameters
table_files:Path (or ID) of the input table to include in the datasetdataset_name:Name of the dataset created for the remediation runrule_set_id:Identifier of the Rule Set to apply during remediation--name:Name of the DNAnexus job-y:Automatically confirms job submission
This execution mode is recommended for repeated remediation workflows, scheduled processing, and production-scale operations.
Last updated