Example 1: hello

To begin, you'll code a "Hello, World!" workflow that captures the output of a command into a file. WDL syntax may look familiar if you know any C-family language like Java or Perl. For example, keywords like workflow and task are used to define blocks of statements contained inside matched curly braces ({}), and variables are defined using a data type like String or File.

In this example, you will:

  • Write a simple workflow in WDL

  • Learn two ways to capture the standard out (STDOUT) of a command block

Getting Started

To see this in action, make a hello directory for your work, and inside that create the file workflow.wdl with the following contents:

version 1.0 

workflow hello_world { 
    input { 
        String name 
    }

    call write_greeting { 
        input: greet_name = name 
    }
}

task write_greeting { 
    input {
        String greet_name 
    }

    command { 
        echo 'Hello, ${greet_name}!' 
    }

    output {
        File outfile = stdout() 
    }
}

  • The version states that the following WDL follows the WDL 1.0 specification.

  • The workflow keyword defines a workflow name. The contents of the workflow are enclosed in matched curly braces.

  • The input block describes the parameters for the workflow.

  • WDL defines several data types you can use to describe an input value. This workflow requires a String parameter called name.

  • call will execute the task named write_greeting. This similar to executing a function in code.

  • The input keyword allows you to pass arguments to the task. The workflow's name input will be passed as greet_name to the task.

  • The task keyword defines a task called write_greeting.

  • The task also defines an input block with a parameter greet_name. It would be fine to call this name because it would not conflict with workflow's name.

  • The command keyword defines a block of shell commands to execute. The block may be denoted with matched curly braces ({}) or triple angle brackets (<<</>>>).

  • The shell command echo prints a salutation to standard out, AKA STDOUT, which is the normal place for output from command-line program to appear. The variable greet_name is interpolated in the string by surrounding it with ~{} or ${} because the command block uses curly braces. When using triple angle brackets, only the first syntax is valid.

  • The output block defines an outfile variable of the type File. The contents of the file is the captured stdout from the command block.

WDL is not whitespace dependent, so indentation is based on your preference.

In the Setup section, you should have installed the miniwdl tool, which can be useful to check the syntax of your WDL. The following command shows the output when there are no problems:

Introduce an error in your WDL to see how the output changes. For instance, change the version to 2.0 and observe the error message:

Or change the call to write_greetings:

Cromwell will also find this error, but the message will be one of literally thousands of lines of output.

Note that miniwdl uses a different parser than dxCompiler, and each has slightly different ideas of what constitutes valid syntax. For example, miniwdl requires commas in between input items but dxCompiler does not. In spite of their differences, I appreciate the concise reporting of errors that miniwdl provides.

Executing WDL locally with Cromwell

To execute this workflow locally using Cromwell, you must first create a JSON file to define the input name. Create a file called inputs.json with the following contents if you'd like to extend salutations to my friend Geoffrey:

Next, run the following command to execute the workflow:

The output will be copious and should include an indication that the command was successful and the output landed in a file in the cromwell-executions directory that was created:

You can use the cat (concatenate) command to see the contents of the file. Be sure to change the file path to the one created by your execution:

Here is another way to write the command block and capture STDOUT to a named file:

  • The command block here uses triple angle brackets to enclose the shell commands.

  • The variable must be interpolated with ~{} because of the triple angle brackets. The Unix redirect operator > is used to send the STDOUT from echo into the file out.txt.

  • The outfile is set to the file out.txt, which is created by the command block.

If you execute this version, the output should show that the file out.txt was created instead of the file stdout:

I can use cat again to verify that the same file was created:

Creating a WDL applet with dxCompiler

Now that you have verified that the workflow runs correctly on your local machine, it's time to compile this onto the DNAnexus platform. First, create a project in your organization and take note of the project ID. I'll demonstrate using the dx command-line interface to create a project called Workflow Test:

All the dx commands will print help documentation if you supply the -h or --help flags. For instance, run dx new project --help.

You can also use the web interface, in which case you should use dx select to switch to the project. Next, use dxCompiler to compile the workflow into a workflows directory in the new project. In the following command, the dxCompiler prints the new workflow ID upon success:

Running a Workflow from the Web Interface

Use the web interface to inspect the new workflow as shown in Figure 1. Click on the info button (an "i" in a circle to the right of the "Run" button) to verify the workflow ID is the same as you see on the command line.

Use the "Run" button in the web interface to launch the applet as shown in Figure 2. As shown in Figue 2, I indicate the applet's outputs should written to the outputs directory.

Click on the "Analysis Inputs" view to specify a name for the greeting. In Figure 3, you see I have selected the name "Jonas."

Click "Start Analysis" to start the workflow. The web interface will show the progress of running the applet as shown in Figure 4.

Figure 5 shows check marks next to each step that has been completed. Click the button to show inputs and outputs, then click on the link to the output file, which may be stdout or out.txt depending on the version of the workflow you compiled.

Click on the output file name to view the contents of the file as shown in Figure 6.

Click on the "Monitor" view to see how long the job lasted and cost as shown in Figure 7.

Running a Workflow from the Command Line

You can also use the dx CLI to run the applet as shown in the following interactive session:

You can also specify the input JSON on the command line as a string or a file. In the following command, I provide the JSON as a string. Also note the use of the -y (yes) flag to have the workflow run without confirmation:

You can also place the JSON into a file like so:

You can execute the workflow with this JSON file as follows:

You may also run the workflow with the -h|--help flag to see how to pass the arguments on the command line:

For instance, you can also launch the app using the following command to greet "Keith":

However you choose to launch the workflow, the new run should be displayed in the "Monitor" view of the web interface. As shown in Figure 8, the new run finished in under 1 minute.

To find out more about the latest run, click on job's name in the run table. As shown in Figure 9, the platform will reuse files from the first run as it sees that nothing has changed. This is called "smart reuse," and you can disable this feature if you like.

You can also use the CLI to view the results of the run with the dx describe command:

Notice in the preceding output that the Output lists file-GFbPkBj0XFYgB7Vj4pF8XXBQ. You can cat the contents of this file with the CLI:

Alternately, you can download the file:

The preceding command should create a new local file called stdout or out.txt, depending on the version of the workflow you compiled. Use the cat command again to verify the contents:

Creating Command Shortcuts Using a Makefile

You can create command-line shortcuts for all the steps of checking and buildingyour workflow by recording them as targets in a Makefile as follows:

GNU Make (or a similar Make program, which you may need to install) can turn the command make local into the listed Cromwell command to run one of the workflow versions. Makefiles are a handy way to document your work and automate your efforts.

Review

You should now be able to do the following:

  • Write a valid WDL workflow that accepts a string input and interpolates that string in a bash command.

  • Capture the standard output of a command block either using the stdout() WDL directive or by redirecting the output of a Unix command to a named file.

  • Define a File type output from a task.

  • Check the syntax of a WDL file using miniwdl.

  • Execute a WDL workflow on your local computer using Cromwell with the inputs defined in a JSON file.

  • Create a new project to contain a workflow.

  • Compile a WDL workflow into a DNAnexus applet using the the dxCompiler.

  • Run an applet using the web interface or the CLI.

  • Inspect the file output of an applet using the web interface or the

    CLI.

  • Download a file from the platform.

  • Use a Makefile to document and automate the steps for building and running a workflow.

In the next section, you'll learn how to accept a file input and launch parallel processes to speed execution of large tasks.

Review

In this chapter, you learned some more WDL functions.

Resources

Full Documentation

To create a support ticket if there are technical issues:

  1. Go to the Help header (same section where Projects and Tools are) inside the platform

  2. Select "Contact Support"

  3. Fill in the Subject and Message to submit a support ticket.

PreviousWDLNextExample 2: Word Count (wc)

Last updated

Was this helpful?