Troubleshooting and FAQs

Common Issues and Resolutions

This section helps you quickly identify and resolve typical challenges encountered when profiling, remediating, and monitoring data.

1. Unexpected Rule Results

   1. Rules may produce unexpected outcomes when conditions are too broad or when source data does not match initial assumptions. Review the rule logic carefully, validate the input data using Data Profiler results, and always perform a dry run to confirm rule behavior before execution.

2. Rules Fail to Apply

   1. Rules can fail when underlying schemas change or required columns are missing. Verify the dataset structure, ensure column names and data types are correct, and update the rule accordingly. If changes are intentional or study-specific, publish a new version of the rule to maintain traceability.

3. No Visible Improvement After Remediation

   1. If remediation outputs do not show the expected data quality improvements, compare pre- and post-remediation metrics in the Data Quality Assessment Report. This helps determine whether the issue lies in the source data itself or in the remediation logic applied.

Performance Considerations

Data Quality Manager jobs run within the DNAnexus platform environment, and performance is influenced by both data volume and compute configuration.

You should select an instance size that aligns with the total size of your input datasets, especially when profiling or remediating multiple datasets in a single job. Larger datasets or more complex rule logic (such as joins or transformations) may require increased memory and compute resources to ensure stable and timely execution.

To optimize performance, consider starting with smaller test runs or subsets of data to validate rules, then scaling up instance size for full production runs. Monitoring job runtime and resource usage over repeated runs can help you identify the most cost-effective configuration for your workflows.

FAQs

1. Do remediation runs overwrite my original data?

   1. No. Remediation outputs are generated as separate datasets. Your original input data remains unchanged.

2. Can I reuse the same rules across multiple studies or datasets?

   1. Yes. Rules are versioned and designed for reuse. You can apply the same rule across different datasets and publish a new version when updates are required.

3. What is the difference between dry run and execution?

   1. A dry run simulates rule application and generates a Data Quality Assessment Report without producing remediated output. Execution applies the rules and generates a new, remediated dataset along with the assessment report.

4. How can I track whether data quality is improving over time?

   1. Use the Data Monitor to review trends, top violated rules, and remediation history across runs. This allows you to measure progress and detect regressions early.