Example 2: Word Count (wc)

You can write the wc applet using Workflow Description Language (WDL), which is a high-level way to define and chain tasks. You will start by defining a single task, which compiles to an applet on the DNAnexus platform.

In this example, you will:

• Write the wc applet using WDL

Introducting WDL

In the bash applet, the inputs, outputs, and runtime specifications are defined in the dxapp.json file, and the code that runs lives in a separate file. WDL combines all of this into a single file. Create a new directory for your work, and then add the following to a file called wc.wdl:

version 1.0 

task wc_wdl { 
    input {
        File input_file 
    }

    command {
        wc ~{input_file} > wc.txt 
    }

    output {
        File outfile = "wc.txt" 
    }

    runtime {
        docker: "ubuntu:20.04" 
    }
}

• There are several versions of WDL, and this indicates the file will use v1.0.

• A task in WDL will compile to an applet in DNAnexus.

• The input block equates to the inputSpec from the previous chapter. Each input value is declared with a WDL type. Here the input is a File.

• The command block contains the bash code that will be executed at runtime.

• The output block equates to the outputSpec from the previous chapter. As with inputs, each output must declare a type.

• The runtime block equates to the runSpec from the previous chapter. Here, you define that the task will use a Docker image of Ubuntu Linux 20.04.

Validating WDL with WOMtool and miniwdl

First, ensure you have a working Java compiler and have installed all the Java Jar files as described in Chapter 1. Use WOMtool to validate the WDL syntax:

$ java -jar ~/womtool.jar validate wc.wdl
Success!

If you installed the Python miniwdl program, you can also use it to check the syntax. The output on success is something like a parse tree:

$ miniwdl check wc.wdl
wc.wdl
    task wc

To demonstrate the output on error, I'll change the word File to Fiel:

$ miniwdl check wc.wdl
(wc.wdl Ln 13 Col 9) Unknown type Fiel
            Fiel outfile = "wc.txt"
            ^^^^^^^^^^^^^^^^^^^^^^^

Here is the equivalent error from WOMtool:

java -jar ~/womtool.jar validate wc.wdl
Failed to process task definition 'wc' (reason 1 of 1):
No struct definition for 'Fiel' found in available structs: []
make: *** [validate] Error 1

The two tools are written in different languages (Java and Python) and have different stringencies of parsing and different ways of reporting errors. You may find it helpful to use both to track down errors.

Compiling a WDL Task into an Applet

First, use dx pwd to check if you are in your wc project; if not, use dx select to change. Now you can use the dxCompiler jar file you downloaded in Chapter 1 to compile the WDL into an applet:

$ java -jar ~/dxCompiler.jar compile wc.wdl
[warning] Project is unspecified...using currently selected project
project-GGyG8K80K9ZKzkX812yY893V
applet-GJ3PxPj0K9Z68x1Y5zK4236B

Run the new applet from the CLI with the help flag to inspect the usage:

$ dx run applet-GJ3PxPj0K9Z68x1Y5zK4236B -h
usage: dx run applet-GJ3PxPj0K9Z68x1Y5zK4236B [-iINPUT_NAME=VALUE ...]

Applet: wc_wdl

Inputs:
  input_file: -iinput_file=(file)

 Reserved for dxCompiler
  overrides___: [-ioverrides___=(hash)]

  overrides______dxfiles: [-ioverrides______dxfiles=(file)
    [-ioverrides______dxfiles=... [...]]]

Outputs:
  outfile: outfile (file)

Whether you use bash or WDL to write an applet, the compiled result works the same for the user.

Running the Applet

If you look in the web interface, you should see a new wc_wdl object in the project as shown in Figure 1.

Click on the applet to launch the user interface as shown in Figure 2. Select an input file and launch the applet.

As with the bash version, you can launch the applet using the command line arguments:

$ dx run applet-GJ3PxPj0K9Z68x1Y5zK4236B \
> -iinput_file=file-GGyG8z00K9Z9GQ9jG4qB4gpX -y --watch

Using input JSON:
{
    "input_file": {
        "$dnanexus_link": "file-GGyG8z00K9Z9GQ9jG4qB4gpX"
    }
}

Calling applet-GJ3PxPj0K9Z68x1Y5zK4236B with output destination
  project-GGyG8K80K9ZKzkX812yY893V:/

Job ID: job-GJ3Q0V80K9Z54K2X9Bzf2v0B

Job Log
-------
Watching job job-GJ3Q0V80K9Z54K2X9Bzf2v0B. Press Ctrl+C to stop watching.

The output from the job will look different, but the result will be the same. You can use dx describe with the --json option to get a JSON document describing the entire job and pipe this to the jq tool to extract the output section:

$ dx describe job-GJ3Q0V80K9Z54K2X9Bzf2v0B --json | jq .output
{
  "outfile": {
    "$dnanexus_link": "file-GJ3Q10Q0b0qvyB6fG7pgx0bX"
  }
}

The dx cat command allows you to quickly see the contents of the output file without having to download it to your computer:

$ dx cat file-GJ3Q10Q0b0qvyB6fG7pgx0bX
  8590  86055 513523 /home/dnanexus/inputs/input1217954139984307828/scarlet.txt

This is the same output as from the previous chapter.

Review

Depending on your comfort level with WDL, you may or may not find this version simpler than the bash version. The result is the same no matter how you write the applet, so it's a matter of taste as to which you should select.

In this chapter, you learned how to:

• Write a WDL task

• Use WOMtool and miniwdl to validate WDL syntax

• Compile a WDL task into an applet

• Use the JSON output from dx describe and jq to extract the outputs of a job

• Use dx cat to see the contents of a file on the DNAnexus platform

Resources

Full Documentation

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform

2. Select "Contact Support"

3. Fill in the Subject and Message to submit a support ticket.