Example 1: hello

To begin, you'll code a "Hello, World!" workflow that captures the output of a command into a file. WDL syntax may look familiar if you know any C-family language like Java or Perl. For example, keywords like workflow and task are used to define blocks of statements contained inside matched curly braces ({}), and variables are defined using a data type like String or File.

In this example, you will:

  • Write a simple workflow in WDL

  • Learn two ways to capture the standard out (STDOUT) of a command block

Getting Started

To see this in action, make a hello directory for your work, and inside that create the file workflow.wdl with the following contents:

version 1.0 

workflow hello_world { 
    input { 
        String name 
    }

    call write_greeting { 
        input: greet_name = name 
    }
}

task write_greeting { 
    input {
        String greet_name 
    }

    command { 
        echo 'Hello, ${greet_name}!' 
    }

    output {
        File outfile = stdout() 
    }
}

  • The version states that the following WDL follows the WDL 1.0 specification.

  • The workflow keyword defines a workflow name. The contents of the workflow are enclosed in matched curly braces.

  • The input block describes the parameters for the workflow.

  • WDL defines several data types you can use to describe an input value. This workflow requires a String parameter called name.

  • call will execute the task named write_greeting. This similar to executing a function in code.

  • The input keyword allows you to pass arguments to the task. The workflow's name input will be passed as greet_name to the task.

  • The task keyword defines a task called write_greeting.

  • The task also defines an input block with a parameter greet_name. It would be fine to call this name because it would not conflict with workflow's name.

  • The command keyword defines a block of shell commands to execute. The block may be denoted with matched curly braces ({}) or triple angle brackets (<<</>>>).

  • The shell command echo prints a salutation to standard out, AKA STDOUT, which is the normal place for output from command-line program to appear. The variable greet_name is interpolated in the string by surrounding it with ~{} or ${} because the command block uses curly braces. When using triple angle brackets, only the first syntax is valid.

  • The output block defines an outfile variable of the type File. The contents of the file is the captured stdout from the command block.

WDL is not whitespace dependent, so indentation is based on your preference.

In the Setup section, you should have installed the miniwdl tool, which can be useful to check the syntax of your WDL. The following command shows the output when there are no problems:

$ miniwdl check workflow.wdl
workflow.wdl
    workflow hello_world
        call write_greeting
    task write_greeting

Introduce an error in your WDL to see how the output changes. For instance, change the version to 2.0 and observe the error message:

$ miniwdl check workflow.wdl
(workflow.wdl Ln 0 Col 0) unknown WDL version 2.0; choices:
draft-2, 1.0, development, 1.1

Or change the call to write_greetings:

$ miniwdl check workflow.wdl
(workflow.wdl Ln 8 Col 5) No such task/workflow: write_greetings
        call write_greetings {
        ^^^^^^^^^^^^^^^^^^^^^^

Cromwell will also find this error, but the message will be one of literally thousands of lines of output.

Note that miniwdl uses a different parser than dxCompiler, and each has slightly different ideas of what constitutes valid syntax. For example, miniwdl requires commas in between input items but dxCompiler does not. In spite of their differences, I appreciate the concise reporting of errors that miniwdl provides.

Executing WDL locally with Cromwell

To execute this workflow locally using Cromwell, you must first create a JSON file to define the input name. Create a file called inputs.json with the following contents if you'd like to extend salutations to my friend Geoffrey:

{ "hello_world.name": "Geoffrey" }

Next, run the following command to execute the workflow:

$ java -jar ~/cromwell-82.jar run --inputs inputs.json workflow.wdl

The output will be copious and should include an indication that the command was successful and the output landed in a file in the cromwell-executions directory that was created:

{
  "hello_world.write_greeting.outfile":
  "/Users/[email protected]/work/srna/wdl_tutorial/hello/
  cromwell-executions/hello_world/7f02fe78-4aff-4e01-95da-c9b6e021773d/
  call-write_greeting/execution/stdout"
}

You can use the cat (concatenate) command to see the contents of the file. Be sure to change the file path to the one created by your execution:

$ cat cromwell-executions/hello_world/7f02fe78-4aff-4e01-95da-c9b6e021773d/call-write_greeting/execution/stdout
Hello, Geoffrey!

Here is another way to write the command block and capture STDOUT to a named file:

version 1.0

workflow hello_world {
    input {
        String name
    }

    call write_greeting {
        input: greet_name = name
    }
}

task write_greeting {
    input {
        String greet_name
    }

    command <<< 
        echo 'Hello, ~{greet_name}!' > out.txt 
    >>>

    output {
        File outfile = "out.txt" 
    }
}

  • The command block here uses triple angle brackets to enclose the shell commands.

  • The variable must be interpolated with ~{} because of the triple angle brackets. The Unix redirect operator > is used to send the STDOUT from echo into the file out.txt.

  • The outfile is set to the file out.txt, which is created by the command block.

If you execute this version, the output should show that the file out.txt was created instead of the file stdout:

{
  "outputs": {
    "hello_world.write_greeting.outfile":
    "/Users/[email protected]/work/srna/wdl_tutorial/hello/
    cromwell-executions/hello_world/1dd3abd8-be70-418b-9a31-b4ea9d5add99/
    call-write_greeting/execution/out.txt"
  },
  "id": "1dd3abd8-be70-418b-9a31-b4ea9d5add99"
}

I can use cat again to verify that the same file was created:

$ cat cromwell-executions/hello_world/1dd3abd8-be70-418b-9a31-b4ea9d5add99/
  call-write_greeting/execution/out.txt
Hello, Geoffrey!

Creating a WDL applet with dxCompiler

Now that you have verified that the workflow runs correctly on your local machine, it's time to compile this onto the DNAnexus platform. First, create a project in your organization and take note of the project ID. I'll demonstrate using the dx command-line interface to create a project called Workflow Test:

$ dx new project "Workflow Test"
Created new project called "Workflow Test" (project-GFbKy7Q0ff1k3fGq48ZFZ45p)
Switch to new project now? [y/N]: y

All the dx commands will print help documentation if you supply the -h or --help flags. For instance, run dx new project --help.

You can also use the web interface, in which case you should use dx select to switch to the project. Next, use dxCompiler to compile the workflow into a workflows directory in the new project. In the following command, the dxCompiler prints the new workflow ID upon success:

$ java -jar ~/dxCompiler-2.10.2.jar compile workflow.wdl -folder /workflows \
> -project project-GFbKy7Q0ff1k3fGq48ZFZ45p
workflow-GFbP9480ff1zVQPG48zXpfzb

Running a Workflow from the Web Interface

Use the web interface to inspect the new workflow as shown in Figure 1. Click on the info button (an "i" in a circle to the right of the "Run" button) to verify the workflow ID is the same as you see on the command line.

Use the "Run" button in the web interface to launch the applet as shown in Figure 2. As shown in Figue 2, I indicate the applet's outputs should written to the outputs directory.

Click on the "Analysis Inputs" view to specify a name for the greeting. In Figure 3, you see I have selected the name "Jonas."

Click "Start Analysis" to start the workflow. The web interface will show the progress of running the applet as shown in Figure 4.

Figure 5 shows check marks next to each step that has been completed. Click the button to show inputs and outputs, then click on the link to the output file, which may be stdout or out.txt depending on the version of the workflow you compiled.

Click on the output file name to view the contents of the file as shown in Figure 6.

Click on the "Monitor" view to see how long the job lasted and cost as shown in Figure 7.

Running a Workflow from the Command Line

You can also use the dx CLI to run the applet as shown in the following interactive session:

$ dx run workflow-GFbP9480ff1zVQPG48zXpfzb
Entering interactive mode for input selection.

Input:   stage-common.name (stage-common.name)
Class:   string

Enter string value ('?' for more options)
stage-common.name: Ronald

Select an optional parameter to set by its # (^D or <ENTER> to finish):

 [0] stage-common.overrides___ (stage-common.overrides___)
 [1] stage-common.overrides______dxfiles (stage-common.overrides______dxfiles)
 [2] stage-0.greet_name (stage-0.greet_name) [default={"$dnanexus_link": {"outputField": "name", "stage": "stage-common"}}]
 [3] stage-0.overrides___ (stage-0.overrides___)
 [4] stage-0.overrides______dxfiles (stage-0.overrides______dxfiles)
 [5] stage-outputs.overrides___ (stage-outputs.overrides___)
 [6] stage-outputs.overrides______dxfiles (stage-outputs.overrides______dxfiles)

Optional param #:
The following 1 stage(s) will reuse results from a previous analysis:
  Stage 2: outputs (job-GFbPJx80ff1gYQy5Fg3pK3GY)


Using input JSON:
{
    "stage-common.name": "Ronald"
}

Confirm running the executable with this input [Y/n]: y
Calling workflow-GFbP9480ff1zVQPG48zXpfzb with output destination
  project-GFbKy7Q0ff1k3fGq48ZFZ45p:/

Analysis ID: analysis-GFbPjVj0ff1ZypqJ8vQj8kjZ

You can also specify the input JSON on the command line as a string or a file. In the following command, I provide the JSON as a string. Also note the use of the -y (yes) flag to have the workflow run without confirmation:

$ dx run workflow-GFbP9480ff1zVQPG48zXpfzb -j '{"stage-common.name": "Ronald"}'
-y
The following 3 stage(s) will reuse results from a previous analysis:
  Stage 0: common (job-GFbPjVj0ff1ZypqJ8vQj8kjf)
  Stage 1: write_greeting (job-GFbPjVj0ff1ZypqJ8vQj8kjg)
  Stage 2: outputs (job-GFbPJx80ff1gYQy5Fg3pK3GY)


Using input JSON:
{
    "stage-common.name": "Ronald"
}

Calling workflow-GFbP9480ff1zVQPG48zXpfzb with output destination
  project-GFbKy7Q0ff1k3fGq48ZFZ45p:/

Analysis ID: analysis-GFbPkFj0ff1k3fGq48ZFZ5Jy

You can also place the JSON into a file like so:

$ cat app_inputs.json
{"stage-common.name": "Ronald"}

You can execute the workflow with this JSON file as follows:

$ dx run -f app_inputs.json workflow-GFbP9480ff1zVQPG48zXpfzb

You may also run the workflow with the -h|--help flag to see how to pass the arguments on the command line:

$ dx run workflow-GFbP9480ff1zVQPG48zXpfzb -h
usage: dx run workflow-GFbP9480ff1zVQPG48zXpfzb [-iINPUT_NAME=VALUE ...]

Workflow: hello_world

Inputs:
 stage-common
  stage-common.name: -istage-common.name=(string)

 stage-common: Reserved for dxCompiler
  stage-common.overrides___: [-istage-common.overrides___=(hash)]

  stage-common.overrides______dxfiles: [-istage-common.overrides______dxfiles=(>

 stage-0
  stage-0.greet_name: [-istage-0.greet_name=(string, default={"$dnanexus_link":>

 stage-0: Reserved for dxCompiler
  stage-0.overrides___: [-istage-0.overrides___=(hash)]

  stage-0.overrides______dxfiles: [-istage-0.overrides______dxfiles=(file) [-is>

 stage-outputs: Reserved for dxCompiler
  stage-outputs.overrides___: [-istage-outputs.overrides___=(hash)]

  stage-outputs.overrides______dxfiles: [-istage-outputs.overrides______dxfiles>

Outputs:
  stage-common.name: stage-common.name (string)

  stage-0.outfile: stage-0.outfile (file)

For instance, you can also launch the app using the following command to greet "Keith":

$ dx run workflow-GFbP9480ff1zVQPG48zXpfzb -istage-common.name=Keith

However you choose to launch the workflow, the new run should be displayed in the "Monitor" view of the web interface. As shown in Figure 8, the new run finished in under 1 minute.

To find out more about the latest run, click on job's name in the run table. As shown in Figure 9, the platform will reuse files from the first run as it sees that nothing has changed. This is called "smart reuse," and you can disable this feature if you like.

You can also use the CLI to view the results of the run with the dx describe command:

Result 1:
ID                    analysis-GFbPjVj0ff1ZypqJ8vQj8kjZ
Class                 analysis
Job name              hello_world
Executable name       hello_world
Project context       project-GFbKy7Q0ff1k3fGq48ZFZ45p
Billed to             org-sos
Workspace             container-GFbPjVj0ff1ZypqJ8vQj8kjb
Workflow              workflow-GFbP9480ff1zVQPG48zXpfzb
Priority              normal
State                 done
Root execution        analysis-GFbPjVj0ff1ZypqJ8vQj8kjZ
Parent job            -
Stage 0               common (stage-common)
  Executable          applet-GFbP93j0ff1py9y87vzB2QQJ
  Execution           job-GFbPjVj0ff1ZypqJ8vQj8kjf (done)
Stage 1               write_greeting (stage-0)
  Executable          applet-GFbP9380ff1XzVKkG9kyVg64
  Execution           job-GFbPjVj0ff1ZypqJ8vQj8kjg (done)
Stage 2               outputs (stage-outputs)
  Executable          applet-GFbP9400ff1pK6v113KJQF9g
  Execution           [job-GFbPJx80ff1gYQy5Fg3pK3GY] (done)
  Cached from         analysis-GFbPJx80ff1gYQy5Fg3pK3GP
Input                 stage-common.name = "Ronald"
                      [stage-0.greet_name = {"$dnanexus_link": {"analysis":
                       "analysis-GFbPjVj0ff1ZypqJ8vQj8kjZ", "stage":
                       "stage-common", "field": "name", "wasInternal": true}}]
Output                stage-common.name = "Ronald"
                      stage-0.outfile = file-GFbPkBj0XFYgB7Vj4pF8XXBQ
Output folder         /
Launched by           kyclark
Created               Wed Aug  3 15:52:55 2022
Finished              Wed Aug  3 15:54:51 2022 (Wall-clock time: 0:01:55)
Last modified         Wed Aug  3 15:54:54 2022
Depends on            -
Tags                  -
Properties            -
Total Price           $0.00
detachedFrom          null
rank                  0
priceComputedAt       1659567291327
currency              {"dxCode": 0, "code": "USD", "symbol": "$",
                      "symbolPosition": "left",
                      "decimalSymbol": ".",
                      "groupingSymbol": ","}
totalEgress           {"regionLocalEgress": 0, "internetEgress": 0,
                      "interRegionEgress": 0}
egressComputedAt      1659567291327
costLimit             null

Notice in the preceding output that the Output lists file-GFbPkBj0XFYgB7Vj4pF8XXBQ. You can cat the contents of this file with the CLI:

$ dx cat file-GFbPkBj0XFYgB7Vj4pF8XXBQ
Hello, Ronald!

Alternately, you can download the file:

$ dx download file-GFbPkBj0XFYgB7Vj4pF8XXBQ
[===========================================================>] Completed 15
of 15 bytes (100%) /Users/[email protected]/work/srna/wdl_tutorial/stdout

The preceding command should create a new local file called stdout or out.txt, depending on the version of the workflow you compiled. Use the cat command again to verify the contents:

$ cat stdout
Hello, Ronald!

Creating Command Shortcuts Using a Makefile

You can create command-line shortcuts for all the steps of checking and buildingyour workflow by recording them as targets in a Makefile as follows:

WORKFLOW = workflow.wdl
PROJECT_ID = project-GFPQvY007GyyXgXGP7x9zbGb
DXCOMPILER = java -jar ~/dxCompiler-2.10.2.jar
CROMWELL = java -jar ~/cromwell-82.jar

check:
    miniwdl check $(WORKFLOW)

local:
    $(CROMWELL) run --inputs inputs.json $(WORKFLOW)

local2:
    $(CROMWELL) run workflow2.wdl

app:
    $(DXCOMPILER) compile $(WORKFLOW) \
        -archive \
        -folder /workflows \
        -project $(PROJECT_ID)

clean:
    rm -rf cromwell-workflow-logs cromwell-executions

GNU Make (or a similar Make program, which you may need to install) can turn the command make local into the listed Cromwell command to run one of the workflow versions. Makefiles are a handy way to document your work and automate your efforts.

Review

You should now be able to do the following:

  • Write a valid WDL workflow that accepts a string input and interpolates that string in a bash command.

  • Capture the standard output of a command block either using the stdout() WDL directive or by redirecting the output of a Unix command to a named file.

  • Define a File type output from a task.

  • Check the syntax of a WDL file using miniwdl.

  • Execute a WDL workflow on your local computer using Cromwell with the inputs defined in a JSON file.

  • Create a new project to contain a workflow.

  • Compile a WDL workflow into a DNAnexus applet using the the dxCompiler.

  • Run an applet using the web interface or the CLI.

  • Inspect the file output of an applet using the web interface or the

    CLI.

  • Download a file from the platform.

  • Use a Makefile to document and automate the steps for building and running a workflow.

In the next section, you'll learn how to accept a file input and launch parallel processes to speed execution of large tasks.

Review

In this chapter, you learned some more WDL functions.

Resources

Full Documentation

To create a support ticket if there are technical issues:

  1. Go to the Help header (same section where Projects and Tools are) inside the platform

  2. Select "Contact Support"

  3. Fill in the Subject and Message to submit a support ticket.

PreviousWDLNextExample 2: Word Count (wc)

Last updated

Was this helpful?