Running nf-core/variantbenchmarking on DNAnexus

Necessary Disclaimers and Legal

The user is responsible for reviewing and complying with the license requirements of the software, and data referenced in this documentation.

Users are responsible for the costs associated with analyzing data, software and its storage in their project spaces.

Instance type availability and pricing are subject to the contract between the user or the user’s organization and DNAnexus.

Citations for nf-core/variantbenchmarking

nf-core is a community-driven collection of curated bioinformatics pipelines built with Nextflow. It provides standardized, scalable, and portable workflows for reproducible analysis across diverse computing environments. nf-core/variantbenchmarking is developed and maintained by the nf-core community. If you use nf-core/variantbenchmarking in your work, please cite the pipeline accordingly. Additional references for integrated tools are listed in the pipeline’s CITATIONS.md.

Overview

nf-core/variantbenchmarking is designed to evaluate and validate the accuracy of variant calling methods in genomic research. It supports benchmarking of small variants (SNVs and INDELs), structural variants (SVs), and copy number variations (CNVs) for both germline and somatic analyses. The pipeline automates the full benchmarking workflow, from VCF normalization and standardization to statistical comparison against truth sets, and produces standardized performance metrics including Precision, Recall, and F1-score. It is compatible with established gold standard truth sets such as Genome in a Bottle (GIAB) and SEQC2.

Please refer to the official document and Github from the nf-core community to learn more about nf-core/variantbenchmarking.

Where to access nf-core/variantbenchmarking

The nf-core/variantbenchmarking folder on DNAnexus is organized as follows:

variantbenchmarking/
├── variantbenchmarking_v1.5.0   # Nextflow applet
├── datasets/                    # Input datasets
├── example_parameter_files/     # Example parameter JSON files
├── example_results/             # Example output results
├── references/                  # Pre-downloaded reference genomes
├── samplesheet/                 # Example samplesheets
├── scripts/                     # Helper scripts
└── soft_config/                 # Resource & tool configuration files

Note on project ID: Throughout this guide, replace project-xxx with your own DNAnexus project ID . Your project ID can be found on the DNAnexus platform under Project Settings, or by running dx env in the dx CLI.

Note on parameter prefixes: Parameters prefixed with -i (e.g. -ioutdir, -inextflow_run_opts) are DNAnexus applet input arguments passed to dx run. Parameters prefixed with -- (e.g. --genome, --fasta) are Nextflow pipeline parameters, typically supplied via a JSON params file or within nextflow_run_opts.

Copying applet and data into a project

To use the datasets and applet, first copy them from the public project into your own project:

1. Create a new project billed to your organization. See Setting Up a Project for instructions.

2. Go to the Resources tab and find the project "Public Datasets AWS US East". Navigate to the variantbenchmarking folder.

3. Select the folders and files you need.

4. Click Copy from the top right menu and select the project you created in Step 1.

5. Go to your project to start running the pipeline.

Note: The example_results/ folder contains output from completed pipeline runs and is provided for reference only.

Sample sheet input

The samplesheet is a .csv file passed to the pipeline via --input. Each row represents one test VCF to be benchmarked.

Minimal Samplesheet (Required Columns Only)

Every samplesheet must contain at minimum three columns: id, caller, and at least one of test_vcf or test_regions.

id,test_vcf,caller
test1,dx://project-xxx:/path/to/test1.vcf.gz,delly
test2,dx://project-xxx:/path/to/test2.vcf.gz,gatk
test3,dx://project-xxx:/path/to/test3.vcf.gz,cnvkit

Column Descriptions

Column	Description
id	Unique identifier for the sample. No spaces allowed. Must be unique per row.
test_vcf	Path to the VCF file to use as benchmarking test input. Accepted formats: .vcf, .vcf.gz, .bcf, .bcf.gz.
test_regions	Path to a regions file to be used for intersection analysis. Either test_vcf or test_regions must be provided.
caller	Name of the variant caller used to generate the test file. There can be more than one test VCF for the same caller. For unknown caller use unknown

This user guide provides a minimal samplesheet to get started. For all optional columns, please refer to the official nf-core/variantbenchmarking usage documentation and the input schema.

Example samplesheets

Several example samplesheets are provided in the samplesheet/ folder on the platform. Samplesheets files staged on the DNAnexus platform use dx:// paths pointing to the Public Dataset AWS East project (project-J3JyY6j030gzQypGpk273241). Users need to replace this project ID with their own before running.

samplesheet/
├── samplesheet_germline_small.csv
├── samplesheet_small_germline_hg38.csv
├── samplesheet_full_small.csv
├── samplesheet_germline_structural.csv
├── samplesheet_sv_germline_hg37.csv
├── samplesheet_somatic_snv.csv
├── samplesheet_somatic_indel.csv
├── samplesheet_full_somatic.csv
└── samplesheet_full_somatic_staging.csv

samplesheet_full_small.csv has input files downloaded directly from NCBI FTP at runtime. These samplesheets can be used as-is without modifying any paths, making them a convenient option for quickly testing the pipeline.

Reference files

nf-core/variantbenchmarking requires a reference genome (FASTA and FAI index) to correctly align and benchmark variants. Pre-downloaded reference files are available on the platform, organized as follows:

references/
├── Ensembl_GRCh37/
├── GATK_GRCh37/
├── GATK_GRCh38/
├── NCBI_GRCh38/
├── UCSC_CHM13/
├── UCSC_hg19/
└── UCSC_hg38/

Understanding the genome parameter

The --genome parameter serves two purposes:

• It tells the pipeline which chromosome naming convention to use in certain preprocessing steps (see filter_contigs in the Technical considerations section).

• If --fasta and --fai are not provided, the pipeline falls back to automatically fetching the corresponding FASTA and index files from Illumina iGenomes.

Note: Specifying the genome version does not mean the pipeline always uses Illumina iGenomes. It is simply a version label. Providing --fasta and --fai explicitly always takes precedence over the iGenomes fallback.

The name of this parameter depends on how you pass it to the pipeline:

Method	Parameter name	Example
dx run CLI	-igenome	-igenome="GRCh38"
Params JSON file	“genome”	"genome": "GRCh38"

This is because the DNAnexus applet exposes the Nextflow --genome parameter as -igenome on the dx run CLI. Both refer to the same underlying pipeline parameter.

Supported values

GRCh37       # Ensembl GRCh37
GRCh38       # NCBI GRCh38
GATK.GRCh37  # GATK bundle GRCh37
GATK.GRCh38  # GATK bundle GRCh38

Additional reference files

For liftover analysis, an additional sequence dictionary file (.dict) is required via --dictionary. If not provided, Picard CreateSequenceDictionary will generate it automatically.

For RTG Tools, a formatted SDF directory is required via --sdf, which will also be generated automatically if missing.

Truth data

Variant benchmarking requires a well-validated truth set as a reference point for evaluating your test samples. The pipeline compares your results against this truth set to compute key performance metrics, including true positives, false positives, false negatives, precision, and recall. For a full list of available truth data parameters, please refer to the official nf-core/variantbenchmarking usage documentation.

Note: If --method concordance is used, --truth_vcf and --truth_id are not required, as concordance analysis compares test VCFs against each other rather than against a truth set. However, if concordance is combined with other benchmarking methods in the same run, truth data remains required for those methods.

Truth data on the DNAnexus platform

Pre-staged truth datasets are available on the platform, eliminating the need to download or upload reference files manually. Two gold standard sources are provided: Genome in a Bottle (GIAB) and SEQC2. Please refer to the respective user guides for detailed information on available files.

Running the pipeline

Minimal test

Use the test configuration (test.config) provided by nf-core/variantbenchmarking to quickly validate the pipeline. This test profile runs on a small dataset of three germline structural variant call sets from the same HG002 Genome in a Bottle sample on chromosome 21, each generated by a different SV caller (Delly, Lumpy, and Manta). The pipeline is configured for a germline structural variant benchmarking run using SVanalyzer against the HG002 GIAB truth set on GRCh37, including preprocessing steps (normalize, deduplicate, filter_contigs).

Run via UI

1. Navigate to your variantbenchmarking folder and click on the variantbenchmarking_v1.5.0 applet, then Run.

1. Provide a Job Name (e.g., variantbenchmarking_test) and choose an output destination. Note: the screenshot shows the public project information, please add your own output location.

1. Under the COMMON section, enter the output directory name (e.g., test_profile) in the outdir box. The pipeline will create a folder with this name inside the destination folder to save all output files.

1. Under the NEXTFLOW OPTIONS section, enter -profile test,docker in the Nextflow Run Options box.

1. Once all inputs are filled in, click Start Analysis to launch the pipeline.

1. Go to the MONITOR tab to check the pipeline execution progress. The Inputs and Outputs section at the bottom summarizes all input parameters and the published output files from the run.

Run via command line

dx run project-xxx:/variantbenchmarking/variantbenchmarking_v1.5.0 \
  -ioutdir='test_profile' \
  -inextflow_run_opts='-profile test,docker' \
  --destination 'project-xxx:/variantbenchmarking/example_results' \
  --priority high \
  --name "variantbenchmarking_test" \
  -y

Please replace project-xxx with your DNAnexus project ID.

Full-size germline test

The test_full profile runs a full-size germline small variant (SNV and INDEL) benchmarking test using real GIAB data downloaded directly from NCBI FTP. It evaluates HG002 variants against the CMRG v1.00 truth set, which focuses on challenging medically relevant genes on GRCh37. Three query VCFs are benchmarked simultaneously, representing different versions of GIAB truth sets (difficult medical genes v0.01, v0.02.03, and v4beta). Four preprocessing steps are applied before benchmarking: split_multiallelic, normalize, deduplicate, and filter_contigs. Two benchmarking tools are run in parallel: hap.py and rtgtools.

dx run project-xxx:/variantbenchmarking/variantbenchmarking_v1.5.0 \
  -ioutdir='test_full_profile' \
  -inextflow_run_opts='-profile test_full,docker' \
  --destination 'project-xxx:/variantbenchmarking/example_results' \
  --priority high \
  --name "variantbenchmarking_test_full" \
  -y

Note: Replace project-xxx with your DNAnexus project ID.

Full-size somatic test

The test_full_somatic profile runs a full-size somatic INDEL benchmarking test on the HCC1395 tumor/normal cell line pair (HCC1395T vs HCC1395N) using GRCh38 as the reference genome. It evaluates INDEL calls from three somatic variant callers (FreeBayes, Mutect2, and Strelka2) against the SEQC2 somatic INDEL superset truth set (v1.1) downloaded directly from NCBI FTP, restricted to high-confidence regions. Four preprocessing steps are applied before benchmarking: split_multiallelic, normalize, deduplicate, and filter_contigs. Two benchmarking tools are run in parallel: som.py and rtgtools.

dx run project-xxx:/variantbenchmarking/variantbenchmarking_v1.5.0 \
  -ioutdir='test_full_somatic_profile' \
  -inextflow_run_opts='-profile test_full_somatic,docker' \
  --destination 'project-xxx:/variantbenchmarking/example_results' \
  --priority high \
  --name "variantbenchmarking_test_full_somatic" \
  -y

Note: Replace project-xxx with your DNAnexus project ID.

Full-size structural variant test

The test_full_sv profile runs a full-size germline structural variant (SV) benchmarking test on HG002 using GRCh37 as the reference genome. It evaluates SV calls from two callers (Manta and Delly) against the GIAB SV Tier1 v0.6 truth set downloaded directly from NCBI FTP, restricted to high-confidence regions. Only one preprocessing step is applied before benchmarking: filter_contigs. Two benchmarking tools are run in parallel: svanalyzer and truvari.

dx run project-xxx:/variantbenchmarking/variantbenchmarking_v1.5.0 \
  -ioutdir='test_full_sv_profile' \
  -inextflow_run_opts='-profile test_full_sv,docker' \
  --destination 'project-xxx:/variantbenchmarking/example_results' \
  --priority high \
  --name "variantbenchmarking_test_full_sv" \
  -y

Note: Replace project-xxx with your DNAnexus project ID.

Running a full job

There are two approaches to submitting a full pipeline run on DNAnexus: using a parameter JSON file, or passing all inputs directly via dx run.

Option 1: Using a parameter JSON file

The parameter JSON file defines all inputs passed to the pipeline, equivalent to -- command-line arguments. Save it on the platform and select it when submitting the job.

The file has four sections: input/output, analysis settings, reference genome, and truth data:

{
  "input": "dx://project-xxx:/variantbenchmarking/samplesheet/samplesheet_full_somatic.csv",
  "outdir": "somatic_indel_SEQC2_GRCh38",

  "analysis": "somatic",
  "variant_type": "indel",
  "method": "sompy,rtgtools",
  "preprocess": "split_multiallelic,normalize,deduplicate,filter_contigs",

  "genome": "GRCh38",
  "fasta": "dx://project-xxx:/variantbenchmarking/references/NCBI_GRCh38/genome.fa",
  "fai": "dx://project-xxx:/variantbenchmarking/references/NCBI_GRCh38/genome.fa.fai",

  "truth_id": "SEQC2",
  "truth_vcf": "dx://project-xxx:/SEQC2/sINDEL.MDKT.superSet.v1.2.vcf.gz",
  "regions_bed": "dx://project-xxx:/SEQC2/High-Confidence_Regions_v1.2.bed"
}

Once the JSON file is saved on the platform, submit the pipeline using dx run:

dx run project-xxx:/variantbenchmarking/variantbenchmarking_v1.5.0 \
  -inextflow_params_file='project-xxx:/path/to/params.json' \
  -inextflow_run_opts='-profile docker' \
  --destination 'project-xxx:/path/to/results' \
  --priority high \
  --name 'job_name' \
  -y

Option 2: Passing all inputs directly via dx run

Alternatively, all parameters can be passed directly as -i inputs to dx run, without a params JSON file. This is useful for quick runs or one-off jobs. The example below shows a full somatic SNV benchmarking run against the SEQC2 truth set on GRCh38:

dx run project-xxx:/variantbenchmarking/variantbenchmarking_v1.5.0 \
  -inextflow_run_opts="-profile docker" \
  -inextflow_soft_confs="project-xxx:/variantbenchmarking/soft_config/VCF_TO_CSV_configs.config" \
  -iinput="project-xxx:/variantbenchmarking/samplesheet/samplesheet_somatic_snv.csv" \
  -ioutdir="somatic_snv_variants" \
  -ianalysis="somatic" \
  -ivariant_type="snv" \
  -igenome="GRCh38" \
  -itruth_vcf="project-xxx:/SEQC2/high-confidence_sSNV_in_HC_regions_v1.2_gt.vcf.gz" \
  -iregions_bed="project-xxx:/SEQC2/High-Confidence_Regions_v1.2.bed" \
  -itruth_id="SEQC2" \
  -ipreprocess="split_multiallelic,normalize,deduplicate,filter_contigs" \
  -iinclude_expression="TYPE=\"snp\"" \
  --destination="project-xxx:/path/to/results" \
  --name="variantbenchmarking_v1.5.0_somatic_snv" \
  -y

Note: When passing file paths via -i inputs, use the format project-xxx:/path/to/file without a dx:// prefix. The --destination flag (without -i prefix) specifies where output files are copied after the job completes, and is a DNAnexus-level argument, not a Nextflow parameter.

Also note, when copying an example samplesheet make sure to update the project id to your own project.

Understanding the output

After a successful run, results are saved to the folder defined by outdir in your parameter file. The top-level structure looks like this:

<outdir>/
├── pipeline_info/
└── <variant_type>/                        # e.g. small, structural, copynumber
    ├── <truth_id>/                        # e.g. HG002, SEQC2
    │   ├── preprocess/
    │   └── stats/
    ├── <sample_id>/                       # e.g. test1, test2, test3
    │   ├── preprocess/
    │   ├── stats/
    │   └── benchmarks/
    │       ├── truvari/
    │       ├── svanalyzer/
    │       ├── wittyer/
    │       ├── rtgtools/
    │       ├── happy/
    │       └── sompy/
    ├── multiqc/
    └── summary/
        ├── comparisons/
        ├── datavzrd/
        ├── plots/
        └── tables/

Please refer to the official document for a full description of each output directory.

Technical considerations

Several soft configuration files are provided in the soft_config/ folder to address common issues encountered when running the pipeline:

soft_config/
├── BCFTOOLS_SPLIT_MULTI_truth_fix.config  # Workaround for GIAB v5.0q truth set
├── TRUVARI_BENCH_resources.config         # Memory override for Truvari
└── VCF_TO_CSV_configs.config              # Resource configuration for VCF_TO_CSV process

These files can be passed to the pipeline via -inextflow_soft_confs. See the sections below for details on when and how to use each file.

Out of memory issue

OOM errors occur when a process exceeds its allocated RAM. For example, when running Truvari with refdist=100000 and chunksize=100000 against GIAB HG002 v5.0q (~30,000+ SVs) on full genome, the process may be killed. Therefore, a soft config file is needed: TRUVARI_BENCH_resources.config to override default memory for the Truvari process:

process {
    withName: 'TRUVARI_BENCH' {
        memory = { 200.GB * task.attempt }
        time   = { 12.h  * task.attempt }
    }
}

Then, pass it to the pipeline via -inextflow_soft_confs.

filter_contigs preprocessing requires --genome

When using filter_contigs as part of the --preprocess option, the --genome parameter must be explicitly specified, even if --fasta and --fai are already provided. This is because the pipeline does not infer the contig list from the .fai file. Instead, it checks whether params.genome contains "38" to determine the correct chromosome naming convention and hardcodes the -r regions argument passed to bcftools view accordingly:

• If --genome contains "38" (e.g. GRCh38, hg38): filters to chr1–chr22, chrX, chrY

• Otherwise (e.g. GRCh37, hg19): filters to 1–22, X, Y

Please refer to modules.config for further information. Omitting --genome will cause the following pipeline error:

Please specify --genome when using filter_contigs for --preprocessing

To resolve this, add --genome (e.g. --genome GRCh38) alongside your --fasta and --fai:

"genome": "GRCh38",
"fasta": "dx://project-xxx:/variantbenchmarking/references/NCBI_GRCh38/genome.fa",
"fai":   "dx://project-xxx:/variantbenchmarking/references/NCBI_GRCh38/genome.fa.fai"

Germline small variant benchmarking with GIAB v5.0q truth set

When using the GIAB v5.0q small variant truth set (HG002_GRCh37_v5.0q_smvar.vcf.gz or HG002_GRCh38_v5.0q_smvar.vcf.gz), the pipeline may fail at the BCFTOOLS_SPLIT_MULTI step in the PREPARE_VCFS_TRUTH subworkflow.

This failure is caused by a known pipeline limitation: bcftools norm performs variant splitting and output indexing in a single command. When processing long INDEL variants, such as variants with an ALT allele length greater than 50 bp in the v5.0q truth set, the indexing step may fail because the output contains out-of-order positions.

This issue does not occur with the GIAB v4.2.1 truth set, which contains fewer long INDEL variants.

Workaround

Pass a soft config file via -inextflow_soft_confs to force BCFTOOLS_SPLIT_MULTI in the PREPARE_VCFS_TRUTH subworkflow to exclude variants with REF or ALT lengths greater than 50 bp before indexing:

process {
    withName: VCF_TO_CSV {
        cpus   = { 2     * task.attempt }
        memory = { 12.GB * task.attempt }
        time   = { 4.h   * task.attempt }
    }
    withName: 'NFCORE_VARIANTBENCHMARKING:VARIANTBENCHMARKING:PREPARE_VCFS_TRUTH:BCFTOOLS_SPLIT_MULTI' {
        ext.args = "--exclude \"STRLEN(REF)>50 || STRLEN(ALT)>50\" --output-type z -m-any -c w --write-index=tbi"
    }
}   

The 50 bp cutoff is consistent with the standard distinction between small variants and structural variants in the literature: variants ≥50 bp are conventionally classified as structural variants and should therefore be benchmarked using the structural variant pipeline with the corresponding truth set (v5.0q_stvar.vcf.gz).

A pre-built soft config file for this workaround is available at:

project-xxx:/variantbenchmarking/soft_config/BCFTOOLS_SPLIT_MULTI_truth_fix.config

Genotype addition for SEQC2 truth data

SEQC2 SNV and INDEL truth data were downloaded from the NCBI SEQC Somatic Mutation WG release directory. A heterozygous genotype value (GT 0/1) was added to each variant to enable compatibility with standard benchmarking tools, as the original files do not contain a FORMAT or sample column. The modified files differ from the originals only in the addition of two columns:

• FORMAT column with value GT

• SEQC2 sample column with value 0/1 for all variants

All other fields (CHROM, POS, ID, REF, ALT, QUAL, FILTER, INFO) remain identical between the original and modified files, as verified by diff comparison.

When truth VCF files without genotype information are used as input, the pipeline fails at the PREPARE_VCFS_TRUTH:BCFTOOLS_REHEADER_TRUTH step with the following error:

missing FORMAT fields, cowardly refusing to add samples

On the DNAnexus platform, two SEQC2 truth VCF files (v1.2) with genotype (GT) information added in SEQC2 folder are available for use:

SEQC2
├── high-confidence_sINDEL_in_HC_regions_v1.2_gt.vcf.gz
├── high-confidence_sSNV_in_HC_regions_v1.2_gt.vcf.gz

If you need to add genotype information to other SEQC2 files in the future, the following helper script can be used:

awk '
  /^##/ { print; next } 
  /^#CHROM/ {
    print "##FORMAT=<ID=GT,Number=1,Type=String,Description=\"Genotype\">";
    print $0 "\tFORMAT\tSEQC2";
    next 
  } 
  { print $0 "\tGT\t0/1" }
' $1 > $2

We also generated two additional files with genotype information (v1.2.1), available in the SEQC2/ folder:

• high-confidence_sINDEL_in_HC_regions_v1.2.1_gt.vcf.gz (1,625 variants)

• high-confidence_sSNV_in_HC_regions_v1.2.1_gt.vcf.gz (39,447 variants)