> For the complete documentation index, see [llms.txt](https://academy.dnanexus.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://academy.dnanexus.com/public-datasets-on-the-dnanexus-platform/validation-ready-accelerator/running-nf-core-variantbenchmarking-on-dnanexus.md). # Running nf-core/variantbenchmarking on DNAnexus ## Necessary Disclaimers and Legal The user is responsible for reviewing and complying with the license requirements of the software, and data referenced in this documentation. Users are responsible for the costs associated with analyzing data, software and its storage in their project spaces. Instance type availability and pricing are subject to the contract between the user or the user’s organization and DNAnexus. Note: Cellranger is a commercial tool from 10X Genomics Inc. and falls under the EULA from 10X Genomics Inc. The container provided for the CellRanger functionality in this pipeline has been built by the nf-core community and is therefore not supported by 10X genomics directly. ## Citations for nf-core/variantbenchmarking nf-core is a community-driven collection of curated bioinformatics pipelines built with Nextflow. It provides standardized, scalable, and portable workflows for reproducible analysis across diverse computing environments. nf-core/variantbenchmarking is developed and maintained by the nf-core community. If you use nf-core/variantbenchmarking in your work, please cite the [pipeline](https://nf-co.re/variantbenchmarking/1.5.0) accordingly. Additional references for integrated tools are listed in the pipeline’s [CITATIONS.md](https://github.com/nf-core/variantbenchmarking/blob/1.5.0/CITATIONS.md). ## Overview nf-core/variantbenchmarking is designed to evaluate and validate the accuracy of variant calling methods in genomic research. It supports benchmarking of small variants (SNVs and INDELs), structural variants (SVs), and copy number variations (CNVs) for both germline and somatic analyses. The pipeline automates the full benchmarking workflow, from VCF normalization and standardization to statistical comparison against truth sets, and produces standardized performance metrics including Precision, Recall, and F1-score. It is compatible with established gold standard truth sets such as Genome in a Bottle (GIAB) and SEQC2. Please refer to the [official document](https://nf-co.re/variantbenchmarking/1.5.0) and [Github ](https://github.com/nf-core/variantbenchmarking/tree/1.5.0)from the nf-core community to learn more about nf-core/variantbenchmarking. ## Where to access nf-core/variantbenchmarking The nf-core/variantbenchmarking folder on DNAnexus is organized as follows: ``` variantbenchmarking/ ├── variantbenchmarking_v1.5.0 # Nextflow applet ├── datasets/ # Input datasets ├── example_parameter_files/ # Example parameter JSON files ├── example_results/ # Example output results ├── references/ # Pre-downloaded reference genomes ├── samplesheet/ # Example samplesheets ├── scripts/ # Helper scripts └── soft_config/ # Resource & tool configuration files ``` Note on project ID: Throughout this guide, replace project-xxx with your own DNAnexus project ID . Your project ID can be found on the DNAnexus platform under Project Settings, or by running dx env in the dx CLI. Note on parameter prefixes: Parameters prefixed with -i (e.g. -ioutdir, -inextflow\_run\_opts) are DNAnexus applet input arguments passed to dx run. Parameters prefixed with -- (e.g. --genome, --fasta) are Nextflow pipeline parameters, typically supplied via a JSON params file or within nextflow\_run\_opts. ## Copying applet and data into a project To use the datasets and applet, first copy them from the public project into your own project: 1. Create a new project billed to your organization. See [Setting Up a Project](https://academy.dnanexus.com/overview-of-the-platform/setting-up-a-project) for instructions. 2. Go to the Resources tab and find the project "Public Datasets AWS US East". Navigate to the variantbenchmarking folder. 3. Select the folders and files you need. 4. Click Copy from the top right menu and select the project you created in Step 1. 5. Go to your project to start running the pipeline. Note: The example\_results/ folder contains output from completed pipeline runs and is provided for reference only. ## Sample sheet input The samplesheet is a .csv file passed to the pipeline via --input. Each row represents one test VCF to be benchmarked. ### Minimal Samplesheet (Required Columns Only) Every samplesheet must contain at minimum three columns: id, caller, and at least one of test\_vcf or test\_regions. ``` id,test_vcf,caller test1,dx://project-xxx:/path/to/test1.vcf.gz,delly test2,dx://project-xxx:/path/to/test2.vcf.gz,gatk test3,dx://project-xxx:/path/to/test3.vcf.gz,cnvkit ``` ### Column Descriptions | Column | Description | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | id | Unique identifier for the sample. No spaces allowed. Must be unique per row. | | test\_vcf | Path to the VCF file to use as benchmarking test input. Accepted formats: .vcf, .vcf.gz, .bcf, .bcf.gz. | | test\_regions | Path to a regions file to be used for intersection analysis. Either test\_vcf or test\_regions must be provided. | | caller | Name of the variant caller used to generate the test file. There can be more than one test VCF for the same caller. For unknown caller use unknown | This user guide provides a minimal samplesheet to get started. For all optional columns, please refer to the official [nf-core/variantbenchmarking usage documentation](https://nf-co.re/variantbenchmarking/1.5.0/docs/usage/) and the [input schema](https://github.com/nf-core/variantbenchmarking/blob/1.5.0/assets/schema_input.json). ### Example samplesheets Several example samplesheets are provided in the samplesheet/ folder on the platform. Samplesheets files staged on the DNAnexus platform use dx:// paths pointing to the Public Dataset AWS East project (project-J3JyY6j030gzQypGpk273241). Users need to replace this project ID with their own before running. ``` samplesheet/ ├── samplesheet_germline_small.csv ├── samplesheet_small_germline_hg38.csv ├── samplesheet_full_small.csv ├── samplesheet_germline_structural.csv ├── samplesheet_sv_germline_hg37.csv ├── samplesheet_somatic_snv.csv ├── samplesheet_somatic_indel.csv ├── samplesheet_full_somatic.csv └── samplesheet_full_somatic_staging.csv ``` samplesheet\_full\_small.csv has input files downloaded directly from NCBI FTP at runtime. These samplesheets can be used as-is without modifying any paths, making them a convenient option for quickly testing the pipeline. ## Reference files nf-core/variantbenchmarking requires a reference genome (FASTA and FAI index) to correctly align and benchmark variants. Pre-downloaded reference files are available on the platform, organized as follows: ``` references/ ├── Ensembl_GRCh37/ ├── GATK_GRCh37/ ├── GATK_GRCh38/ ├── NCBI_GRCh38/ ├── UCSC_CHM13/ ├── UCSC_hg19/ └── UCSC_hg38/ ``` ### Understanding the genome parameter The --genome parameter serves two purposes: * It tells the pipeline which chromosome naming convention to use in certain preprocessing steps (see filter\_contigs in the Technical considerations section). * If --fasta and --fai are not provided, the pipeline falls back to automatically fetching the corresponding FASTA and index files from Illumina iGenomes. Note: Specifying the genome version does not mean the pipeline always uses Illumina iGenomes. It is simply a version label. Providing --fasta and --fai explicitly always takes precedence over the iGenomes fallback. The name of this parameter depends on how you pass it to the pipeline: | Method | Parameter name | Example | | ---------------- | -------------- | ------------------ | | dx run CLI | -igenome | -igenome="GRCh38" | | Params JSON file | “genome” | "genome": "GRCh38" | This is because the DNAnexus applet exposes the Nextflow --genome parameter as -igenome on the dx run CLI. Both refer to the same underlying pipeline parameter. #### Supported values ``` GRCh37 # Ensembl GRCh37 GRCh38 # NCBI GRCh38 GATK.GRCh37 # GATK bundle GRCh37 GATK.GRCh38 # GATK bundle GRCh38 ``` #### Additional reference files For liftover analysis, an additional sequence dictionary file (.dict) is required via --dictionary. If not provided, Picard CreateSequenceDictionary will generate it automatically. For RTG Tools, a formatted SDF directory is required via --sdf, which will also be generated automatically if missing. ## Truth data Variant benchmarking requires a well-validated truth set as a reference point for evaluating your test samples. The pipeline compares your results against this truth set to compute key performance metrics, including true positives, false positives, false negatives, precision, and recall. For a full list of available truth data parameters, please refer to the official [nf-core/variantbenchmarking usage documentation](https://nf-co.re/variantbenchmarking/1.5.0/docs/usage/). {% hint style="info" %} Note: If --method concordance is used, --truth\_vcf and --truth\_id are not required, as concordance analysis compares test VCFs against each other rather than against a truth set. However, if concordance is combined with other benchmarking methods in the same run, truth data remains required for those methods. {% endhint %} ### Truth data on the DNAnexus platform Pre-staged truth datasets are available on the platform, eliminating the need to download or upload reference files manually. Two gold standard sources are provided: Genome in a Bottle (GIAB) and SEQC2. Please refer to the respective user guides for detailed information on available files. ## Running the pipeline ### Minimal test Use the test configuration (test.config) provided by nf-core/variantbenchmarking to quickly validate the pipeline. This test profile runs on a small dataset of three germline structural variant call sets from the same HG002 Genome in a Bottle sample on chromosome 21, each generated by a different SV caller (Delly, Lumpy, and Manta). The pipeline is configured for a germline structural variant benchmarking run using SVanalyzer against the HG002 GIAB truth set on GRCh37, including preprocessing steps (normalize, deduplicate, filter\_contigs). #### Run via UI 1. Navigate to your variantbenchmarking folder and click on the variantbenchmarking\_v1.5.0 applet, then Run.
2. Provide a Job Name (e.g., variantbenchmarking\_test) and choose an output destination. Note: the screenshot shows the public project information, please add your own output location.
3. Under the COMMON section, enter the output directory name (e.g., test\_profile) in the outdir box. The pipeline will create a folder with this name inside the destination folder to save all output files. 4. Under the NEXTFLOW OPTIONS section, enter -profile test,docker in the Nextflow Run Options box. 5. Once all inputs are filled in, click Start Analysis to launch the pipeline. 6. Go to the MONITOR tab to check the pipeline execution progress. The Inputs and Outputs section at the bottom summarizes all input parameters and the published output files from the run. #### Run via command line ``` dx run project-xxx:/variantbenchmarking/variantbenchmarking_v1.5.0 \ -ioutdir='test_profile' \ -inextflow_run_opts='-profile test,docker' \ --destination 'project-xxx:/variantbenchmarking/example_results' \ --priority high \ --name "variantbenchmarking_test" \ -y ``` Please replace project-xxx with your DNAnexus project ID. ### Full-size germline test The test\_full profile runs a full-size germline small variant (SNV and INDEL) benchmarking test using real GIAB data downloaded directly from NCBI FTP. It evaluates HG002 variants against the CMRG v1.00 truth set, which focuses on challenging medically relevant genes on GRCh37. Three query VCFs are benchmarked simultaneously, representing different versions of GIAB truth sets (difficult medical genes v0.01, v0.02.03, and v4beta). Four preprocessing steps are applied before benchmarking: split\_multiallelic, normalize, deduplicate, and filter\_contigs. Two benchmarking tools are run in parallel: hap.py and rtgtools. ``` dx run project-xxx:/variantbenchmarking/variantbenchmarking_v1.5.0 \ -ioutdir='test_full_profile' \ -inextflow_run_opts='-profile test_full,docker' \ --destination 'project-xxx:/variantbenchmarking/example_results' \ --priority high \ --name "variantbenchmarking_test_full" \ -y ``` Note: Replace project-xxx with your DNAnexus project ID. ### Full-size somatic test The test\_full\_somatic profile runs a full-size somatic INDEL benchmarking test on the HCC1395 tumor/normal cell line pair (HCC1395T vs HCC1395N) using GRCh38 as the reference genome. It evaluates INDEL calls from three somatic variant callers (FreeBayes, Mutect2, and Strelka2) against the SEQC2 somatic INDEL superset truth set (v1.1) downloaded directly from NCBI FTP, restricted to high-confidence regions. Four preprocessing steps are applied before benchmarking: split\_multiallelic, normalize, deduplicate, and filter\_contigs. Two benchmarking tools are run in parallel: som.py and rtgtools. ``` dx run project-xxx:/variantbenchmarking/variantbenchmarking_v1.5.0 \ -ioutdir='test_full_somatic_profile' \ -inextflow_run_opts='-profile test_full_somatic,docker' \ --destination 'project-xxx:/variantbenchmarking/example_results' \ --priority high \ --name "variantbenchmarking_test_full_somatic" \ -y ``` Note: Replace project-xxx with your DNAnexus project ID. ### Full-size structural variant test The test\_full\_sv profile runs a full-size germline structural variant (SV) benchmarking test on HG002 using GRCh37 as the reference genome. It evaluates SV calls from two callers (Manta and Delly) against the GIAB SV Tier1 v0.6 truth set downloaded directly from NCBI FTP, restricted to high-confidence regions. Only one preprocessing step is applied before benchmarking: filter\_contigs. Two benchmarking tools are run in parallel: svanalyzer and truvari. ``` dx run project-xxx:/variantbenchmarking/variantbenchmarking_v1.5.0 \ -ioutdir='test_full_sv_profile' \ -inextflow_run_opts='-profile test_full_sv,docker' \ --destination 'project-xxx:/variantbenchmarking/example_results' \ --priority high \ --name "variantbenchmarking_test_full_sv" \ -y ``` Note: Replace project-xxx with your DNAnexus project ID. ### Running a full job There are two approaches to submitting a full pipeline run on DNAnexus: using a parameter JSON file, or passing all inputs directly via dx run. #### Option 1: Using a parameter JSON file The parameter JSON file defines all inputs passed to the pipeline, equivalent to -- command-line arguments. Save it on the platform and select it when submitting the job. The file has four sections: input/output, analysis settings, reference genome, and truth data: ``` { "input": "dx://project-xxx:/variantbenchmarking/samplesheet/samplesheet_full_somatic.csv", "outdir": "somatic_indel_SEQC2_GRCh38", "analysis": "somatic", "variant_type": "indel", "method": "sompy,rtgtools", "preprocess": "split_multiallelic,normalize,deduplicate,filter_contigs", "genome": "GRCh38", "fasta": "dx://project-xxx:/variantbenchmarking/references/NCBI_GRCh38/genome.fa", "fai": "dx://project-xxx:/variantbenchmarking/references/NCBI_GRCh38/genome.fa.fai", "truth_id": "SEQC2", "truth_vcf": "dx://project-xxx:/SEQC2/sINDEL.MDKT.superSet.v1.2.vcf.gz", "regions_bed": "dx://project-xxx:/SEQC2/High-Confidence_Regions_v1.2.bed" } ``` Once the JSON file is saved on the platform, submit the pipeline using dx run: ``` dx run project-xxx:/variantbenchmarking/variantbenchmarking_v1.5.0 \ -inextflow_params_file='project-xxx:/path/to/params.json' \ -inextflow_run_opts='-profile docker' \ --destination 'project-xxx:/path/to/results' \ --priority high \ --name 'job_name' \ -y ``` #### Option 2: Passing all inputs directly via dx run Alternatively, all parameters can be passed directly as -i inputs to dx run, without a params JSON file. This is useful for quick runs or one-off jobs. The example below shows a full somatic SNV benchmarking run against the SEQC2 truth set on GRCh38: ``` dx run project-xxx:/variantbenchmarking/variantbenchmarking_v1.5.0 \ -inextflow_run_opts="-profile docker" \ -inextflow_soft_confs="project-xxx:/variantbenchmarking/soft_config/VCF_TO_CSV_configs.config" \ -iinput="project-xxx:/variantbenchmarking/samplesheet/samplesheet_somatic_snv.csv" \ -ioutdir="somatic_snv_variants" \ -ianalysis="somatic" \ -ivariant_type="snv" \ -igenome="GRCh38" \ -itruth_vcf="project-xxx:/SEQC2/high-confidence_sSNV_in_HC_regions_v1.2_gt.vcf.gz" \ -iregions_bed="project-xxx:/SEQC2/High-Confidence_Regions_v1.2.bed" \ -itruth_id="SEQC2" \ -ipreprocess="split_multiallelic,normalize,deduplicate,filter_contigs" \ -iinclude_expression="TYPE=\"snp\"" \ --destination="project-xxx:/path/to/results" \ --name="variantbenchmarking_v1.5.0_somatic_snv" \ -y ``` {% hint style="info" %} Note: When passing file paths via -i inputs, use the format project-xxx:/path/to/file without a dx:// prefix. The --destination flag (without -i prefix) specifies where output files are copied after the job completes, and is a DNAnexus-level argument, not a Nextflow parameter. {% endhint %} Also note, when copying an example samplesheet make sure to update the project id to your own project. ## Understanding the output After a successful run, results are saved to the folder defined by outdir in your parameter file. The top-level structure looks like this: ``` / ├── pipeline_info/ └── / # e.g. small, structural, copynumber ├── / # e.g. HG002, SEQC2 │ ├── preprocess/ │ └── stats/ ├── / # e.g. test1, test2, test3 │ ├── preprocess/ │ ├── stats/ │ └── benchmarks/ │ ├── truvari/ │ ├── svanalyzer/ │ ├── wittyer/ │ ├── rtgtools/ │ ├── happy/ │ └── sompy/ ├── multiqc/ └── summary/ ├── comparisons/ ├── datavzrd/ ├── plots/ └── tables/ ``` Please refer to [the official document](https://nf-co.re/variantbenchmarking/1.5.0/docs/output/) for a full description of each output directory. ## Technical considerations Several soft configuration files are provided in the soft\_config/ folder to address common issues encountered when running the pipeline: ``` soft_config/ ├── BCFTOOLS_SPLIT_MULTI_truth_fix.config # Workaround for GIAB v5.0q truth set ├── TRUVARI_BENCH_resources.config # Memory override for Truvari └── VCF_TO_CSV_configs.config # Resource configuration for VCF_TO_CSV process ``` These files can be passed to the pipeline via -inextflow\_soft\_confs. See the sections below for details on when and how to use each file. ### Out of memory issue OOM errors occur when a process exceeds its allocated RAM. For example, when running Truvari with refdist=100000 and chunksize=100000 against GIAB HG002 v5.0q (\~30,000+ SVs) on full genome, the process may be killed. Therefore, a soft config file is needed: TRUVARI\_BENCH\_resources.config to override default memory for the Truvari process: ``` process { withName: 'TRUVARI_BENCH' { memory = { 200.GB * task.attempt } time = { 12.h * task.attempt } } } ``` Then, pass it to the pipeline via -inextflow\_soft\_confs. ### filter\_contigs preprocessing requires --genome When using filter\_contigs as part of the --preprocess option, the --genome parameter must be explicitly specified, even if --fasta and --fai are already provided. This is because the pipeline does not infer the contig list from the .fai file. Instead, it checks whether params.genome contains "38" to determine the correct chromosome naming convention and hardcodes the -r regions argument passed to bcftools view accordingly: * If --genome contains "38" (e.g. GRCh38, hg38): filters to chr1–chr22, chrX, chrY * Otherwise (e.g. GRCh37, hg19): filters to 1–22, X, Y Please refer to [modules.config](https://github.com/nf-core/variantbenchmarking/blob/1.5.0/conf/modules.config) for further information. Omitting --genome will cause the following pipeline error: `Please specify --genome when using filter_contigs for --preprocessing` To resolve this, add --genome (e.g. --genome GRCh38) alongside your --fasta and --fai: ``` "genome": "GRCh38", "fasta": "dx://project-xxx:/variantbenchmarking/references/NCBI_GRCh38/genome.fa", "fai": "dx://project-xxx:/variantbenchmarking/references/NCBI_GRCh38/genome.fa.fai" ``` ### Germline small variant benchmarking with GIAB v5.0q truth set When using the GIAB v5.0q small variant truth set (HG002\_GRCh37\_v5.0q\_smvar.vcf.gz or HG002\_GRCh38\_v5.0q\_smvar.vcf.gz), the pipeline may fail at the BCFTOOLS\_SPLIT\_MULTI step in the PREPARE\_VCFS\_TRUTH subworkflow. This failure is caused by a known pipeline limitation: bcftools norm performs variant splitting and output indexing in a single command. When processing long INDEL variants, such as variants with an ALT allele length greater than 50 bp in the v5.0q truth set, the indexing step may fail because the output contains out-of-order positions. This issue does not occur with the GIAB v4.2.1 truth set, which contains fewer long INDEL variants. #### Workaround Pass a soft config file via -inextflow\_soft\_confs to force BCFTOOLS\_SPLIT\_MULTI in the PREPARE\_VCFS\_TRUTH subworkflow to exclude variants with REF or ALT lengths greater than 50 bp before indexing: ``` process { withName: VCF_TO_CSV { cpus = { 2 * task.attempt } memory = { 12.GB * task.attempt } time = { 4.h * task.attempt } } withName: 'NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:PREPARE_VCFS_TRUTH:BCFTOOLS_SPLIT_MULTI' { ext.args = "--exclude \"STRLEN(REF)>50 || STRLEN(ALT)>50\" --output-type z -m-any -c w --write-index=tbi" } } ``` The 50 bp cutoff is consistent with the standard distinction between small variants and structural variants in the [literature](https://www.nature.com/articles/s41439-024-00276-x): variants ≥50 bp are conventionally classified as structural variants and should therefore be benchmarked using the structural variant pipeline with the corresponding truth set (v5.0q\_stvar.vcf.gz). A pre-built soft config file for this workaround is available at: ``` project-xxx:/variantbenchmarking/soft_config/BCFTOOLS_SPLIT_MULTI_truth_fix.config ``` ### Genotype addition for SEQC2 truth data SEQC2 SNV and INDEL truth data were downloaded from the NCBI SEQC Somatic Mutation WG release directory. A heterozygous genotype value (GT 0/1) was added to each variant to enable compatibility with standard benchmarking tools, as the original files do not contain a FORMAT or sample column. The modified files differ from the originals only in the addition of two columns: * FORMAT column with value GT * SEQC2 sample column with value 0/1 for all variants All other fields (CHROM, POS, ID, REF, ALT, QUAL, FILTER, INFO) remain identical between the original and modified files, as verified by diff comparison. When truth VCF files without genotype information are used as input, the pipeline fails at the PREPARE\_VCFS\_TRUTH:BCFTOOLS\_REHEADER\_TRUTH step with the following error: ``` missing FORMAT fields, cowardly refusing to add samples ``` On the DNAnexus platform, two SEQC2 truth VCF files (v1.2) with genotype (GT) information added in SEQC2 folder are available for use: ``` SEQC2 ├── high-confidence_sINDEL_in_HC_regions_v1.2_gt.vcf.gz ├── high-confidence_sSNV_in_HC_regions_v1.2_gt.vcf.gz ``` If you need to add genotype information to other SEQC2 files in the future, the following helper script can be used: ``` awk ' /^##/ { print; next } /^#CHROM/ { print "##FORMAT="; print $0 "\tFORMAT\tSEQC2"; next } { print $0 "\tGT\t0/1" } ' $1 > $2 ``` We also generated two additional files with genotype information (v1.2.1), available in the SEQC2/ folder: * high-confidence\_sINDEL\_in\_HC\_regions\_v1.2.1\_gt.vcf.gz (1,625 variants) * high-confidence\_sSNV\_in\_HC\_regions\_v1.2.1\_gt.vcf.gz (39,447 variants)