Example 5: workflow
In this example, you will learn:
How to to accept a BAM file as a workflow input
Break the BAM into slices by chromosome
Distribute the slices in parallel to count the number of alignments in each
Getting Started
To begin, create a new directory called view_and_count and a workflow.wdl file.
Here is the workflow
defintion you should add:
version 1.0
workflow bam_chrom_counter {
input {
File bam
}
String docker_img = "quay.io/biocontainers/samtools:1.12--hd5e65b6_0"
call slice_bam {
input : bam = bam,
docker_img = docker_img
}
scatter (slice in slice_bam.slices) {
call count_bam {
input: bam = slice,
docker_img = docker_img
}
}
output {
File bai = slice_bam.bai
Array[Int] count = count_bam.count
}
}
The name of this workflow is bam_chrom_counter.
The workflow accepts a single, required
File
input that will be calledbam
as it is expected to be a BAM file.Use a non-input declaration to define a
String
value of the Docker file containing Samtools.The first
call
will be to theslice_bam
task that will break the BAM into one file per chromosome. The input for this task is the workflow's BAM file.The
scatter
directive in WDL causes the actions in the block to be executed in parallel, which can lead to significant performance gains. Here, the eachslice
file returned from theslice_bam
task will be used as the input to thecount_bam
task.The workflow defines two outputs: a BAM index file and an array of integer values representing the number of alignments in each of the BAM slices.
Following is the slice_bam
task that uses Samtools to index the input BAM file and break it into separate files for each of the 22 human chromosomes:
task slice_bam {
input {
File bam
String docker_img
}
command <<<
set -ex
samtools index "~{bam}"
mkdir slices
for i in $(seq 22); do
samtools view -b -o "slices/$i.bam" "~{bam}" "chr${i}"
done
>>>
runtime {
docker: docker_img
}
output {
File bai = "~{bam}.bai"
Array[File] slices = glob("slices/*.bam")
}
}
The inputs to this task are the BAM file and the name of the Docker image.
The command block uses triple-angle brackets because it must use the dollar sign (
$
) in shell code.Use
samtools index
on the input BAM file for fast random access to the alignments.The
$()
syntax in bash calls theseq
function to create a sequence of integer values up the 22 human non-sex chromosomes.The
samtools view
will display the alignments in BAM format for a region like "chr1" and place the output into the file slices/1.bam. Note the mix of~
for WDL variables and$
for bash variables.The
runtime
block allows you to define a Docker image that contains an installation of Samtools.The output of this task is the BAM index, which is the given BAM file plus the suffix .bai, and the sliced alignment files.
The
slices
will be one or more files as indicated byArray[File]
, and they will be found using theglob
function to look in the slices directory for all files with the extension .bam.
The count_bam
task is written to handle just one BAM slice:
task count_bam {
input {
File bam
String docker_img
}
command <<<
samtools view -c "~{bam}"
>>>
runtime {
docker: docker_img
}
output {
Int count = read_int(stdout())
}
}
This BAM input will be a slice of alignments for a given region. Naming this
bam
does not interfere with thebam
variable in the workflow or any other task.Use the
samtools view
command with-c|--count
to count the number of alignments in the given file.The output of this task uses the
read_int
function to read theSTDOUT
from the command as an integer value.
At this point, I like to use miniwdl
to check the syntax:
$ miniwdl check workflow.wdl
workflow.wdl
workflow bam_chrom_counter
call slice_bam
scatter slice
call count_bam
task count_bam
task slice_bam
As no errors are reported, I will compile this onto the DNAnexus platform:
$ java -jar ~/dxCompiler-2.10.2.jar compile workflow.wdl \
-archive \
-folder /workflows \
-project project-GFPQvY007GyyXgXGP7x9zbGb
workflow-GFqF27j07GyZ33JX4vzqgK32
Finally, I will run this workflow using a sample BAM file:
$ dx run workflow-GFqF27j07GyZ33JX4vzqgK32 \
> -istage-common.bam=file-G8V38KQ0zQ713kZGF6xQQvjJ -y
Using input JSON:
{
"stage-common.bam": {
"$dnanexus_link": "file-G8V38KQ0zQ713kZGF6xQQvjJ"
}
}
Calling workflow-GFqF27j07GyZ33JX4vzqgK32 with output destination
project-GFPQvY007GyyXgXGP7x9zbGb:/
Analysis ID: analysis-GFqF7Zj07GyZQ957Jy822gQY
Return to the DNAnexus website to monitor the progress of the analysis.
Placing Task Definitions in Files
As the number of tasks increase, workflow definitions can get quite long. You can shorten the workflow.wdl by placing each task in a separate file, which also makes it easier to reuse a task in a separate workflow. To do this, create a subdirectory called tasks, and then create a file called tasks/slice_bam.wdl with the following contents:
version 1.0
task slice_bam {
input {
File bam
String docker_img
}
command <<<
set -ex
samtools index "~{bam}"
mkdir slices
for i in $(seq 22); do
samtools view -b -o "slices/$i.bam" "~{bam}" "chr${i}"
done
>>>
runtime {
docker: docker_img
}
output {
File bai = "~{bam}.bai"
Array[File] slices = glob("slices/*.bam")
}
}
Also create the file tasks/count_bam.wdl with the following contents:
version 1.0
task count_bam {
input {
File bam
String docker_img
}
command <<<
samtools view -c "~{bam}"
>>>
runtime {
docker: docker_img
}
output {
Int count = read_int(stdout())
}
}
Both of the preceding tasks are identical to the original definitions, but note that the files include a version
that matches the version of the workflow. Change workflow.wdl as follows:
version 1.0
import "./tasks/slice_bam.wdl" as task_slice_bam
import "./tasks/count_bam.wdl" as task_count_bam
workflow bam_chrom_counter {
input {
File bam
}
String docker_img = "quay.io/biocontainers/samtools:1.12--hd5e65b6_0"
call task_slice_bam.slice_bam as slice_bam {
input : bam = bam,
docker_img = docker_img
}
scatter (slice in slice_bam.slices) {
call task_count_bam.count_bam as count_bam {
input: bam = slice,
docker_img = docker_img
}
}
output {
File bai = slice_bam.bai
Array[Int] count = count_bam.count
}
}
Use
import
to include WDL code from a file or URI. Note the use of theas
clause to alias the imports using a different name.Call
task_slice_bam.slice_bam
from the imported file usingas
to give it the same name as in the original workflow.Do the same with
task_count_bam.count_bam
.
Use miniwdl
to check your syntax, then use dxCompiler to create an app.
Review
In this lesson, you learned how to:
Accept a file as a workflow input
Define a non-input declaration
Use
scatter
to run tasks in parallelUse the output from one task as the input to another task
Mix
~
and$
in command blocks to dereference WDL and shell variablesImport WDL from external sources such as local files or remote URIs.
Resources
To create a support ticket if there are technical issues:
Go to the Help header (same section where Projects and Tools are) inside the platform
Select "Contact Support"
Fill in the Subject and Message to submit a support ticket.
Last updated
Was this helpful?