Welcome to DNAnexus Academy's online guidebook! This resource is designed for educational purposes to provide you with a foundational understanding of how to utilize DNAnexus for performing analyses. Please note that this guide does not aim to instruct you on every aspect of using the platform, nor does it suggest that this is the only method for leveraging DNAnexus solutions. Instead, it serves as an instructional tool with examples designed to help you begin your journey.

Included in this documentation are guides to assist with your projects, including videos, and content for the terms and concepts that we think are important for your understanding. There are also walk-through examples to get you comfortable on the platform.

As-Is Software Disclaimer This content in this repository is delivered “As-Is”. Notwithstanding anything to the contrary, DNAnexus will have no warranty, support, liability or other obligations with respect to materials provided hereunder.

If you are new to the DNAnexus platform and computational biology/ bioinformatics, these sections are recommended for you:

Welcome to DNAnexus!

Before you go through the information here, there is necessary information that we think will be useful for you to have.

Some of the users of the platform have limited coding experience. As bioinformaticians and computational biologists, we are members of a community that want to help alleviate that stress. On this page, we attached some helpful links and tutorials that will hopefully help make the world of computational biology a bit less intimidating. This is not a partnership or affiliation, but rather a list of what we found useful when we were learning ourselves.

Additionally, users may need resources on the different types of sequencing and the impacts, and we have some here for the ever evolving field of genetics/ genomics. Again, these are not endorsing any particular company, lab, or resource, but instead more of a general guide to help fill in the gaps.

Instance Type Overview

Naming

AWS naming of instance types is broken down here:

Please note, the data present in this page is synthetic data, and is intended for training purposes only. Information about the data present in this documentation is listed.

When germline variant data is present in your data ingestion for the cohort, the Germline Variants tab will appear in the Cohort Browser. The goal of viewing data within the Germline Variants tab is to view germline mutations in genes or genomic regions of interest.

Features of the Germline Variants Tab

To filter with phenotypic data, you can filter from the tiles that you added in the “Overview” tab, or through the “+ Add Filter” button in the Cohort Banner. These filters allow for assessing the impact of phenotypic/ clinical data and the creation of cohorts.

How to Filter a Dataset with Phenotypic Filters

1. In the Cohort section, select the “+ Add Filter”

2. Search or select your characteristic. Ex: Diagnoses, Tumor Details > Tumor Disease Anatomic Site

If you are a Titan user, these sections are recommended for you:

Any background information that could be necessary are listed in the For HPC or For Scientists pages to get you started there as well.

If you are an HPC user new to the DNAnexus platform , these sections are recommended for you:

If you are an experienced user new to the DNAnexus platform, these sections are recommended for you:

If you are an Apollo user, these sections are recommended for you:

Any background information that could be necessary are listed in the For HPC or For Scientists pages to get you started there as well.

In this section, you will build the same applet examples from bash and Python as tasks, and then graduate to building workflows by chaining tasks together.

Workflows are a set of 2 or more apps that are linked together by dependencies, or when the output of one app/ applet is the input to another app/applet. A workflow will allow for these apps to be ran after the dependencies are met without having to submit another job (unless there is an error).

We support the following options for building workflows: * Native (GUI) * WDL * Nextflow

In order to kill a job/ workflow/ app/applet you will need to terminate the job/ analysis. Please use dx terminate or terminate in the Monitor tab in the UI.

All the same examples from bash now in Python.

Please note, the data present in this page is intended for training purposes only. Information about the data present in this documentation is listed here.

The overview tab is dedicated to the phenotypic data that has been ingested in your dataset. The phenotypic data is able to be displayed utilizing tiles, and these tiles will have different tables or figures based on their data type.

Adding Tiles

Simple Tiles

1. Open Cohort Browser

2. Select "+ Add Tile" on the top right corner

1. Find the characteristic you want as a tile and select "Add Tile"

1. Repeat until you select the amount of tiles that you are wanting (up to 15)

2D Plots

• Used for more advanced comparisons

• Add comparisons by selecting the first filter, then selecting the "+" sign for a secondary field

• Then, edit the data field details

Here is the overview of the 2D plots that are available based on data types:

Steps to Create 2D plots

1. Open Cohort Browser

2. Select Add tile on the top right corner

1. Find the characteristic you are wanting to start with and select it, such as biological sex. This is the same step as adding a regular tile, but you will NOT select Add tile.

1. Instead, add a secondary field by selecting this next to the second characteristic you are wanting to view.

1. You will then have options to change the graph with those parameters.

1. Then, select the add tile button on the bottom right below the new graph. This will add it to the cohort browser.

Limits on the Cohort Browser

• Limited to 15 tiles overall in dashboard

• Limited to 30 columns in Data Preview

• Add 1-2 tiles at a time, wait for them to refresh before adding more tiles.

Billing

Definition

Billing occurs monthly based on your use of the platform. These invoices are received at the end of the month

The relationship of DNAnexus and billing are highlighted here:

Billing and Charges

What are the charges?

• Regions and Pricing can be referred to as the "Rate Card"

• These are negotiated at the time of signing

• This is the area of expertise of the DNAnexus Sales Account Director. For further details on this, please refer to them.

• This can be useful for everyone else to decide the instances that you choose to run on the platform.

Errors and Billing

• Job Errors happen

  • Some of which are charged to you

  • Some of which are not

• Error details are found in our

Organizations and Billing

Example: Orgs and Billing

• Orgs can be used to consolidate and simplify billing.

• An order can be associated with a billing account. This allows all users of the org to build projects and apps to the org billing account.

• If you have a project bill to an org, this is useful. Say if you have users within a group or users within a particular lab, that are working with a shared budget, where each member needs to have the ability to work independently within their own project.

• By associating a billing account with an org, this allows groups with a shared budget to consolidate all platform activities onto one invoice.

Resources

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform

2. Select "Contact Support"

3. Fill in the Subject and Message to submit a support ticket.

Basic Concept and Terminology

Key Players in Understanding Cloud Computing

• Your Computer: When we utilize cloud resources, we as users request them from our own computer using commands from the dx toolkit.

• DNAnexus platform: The platform has many working pieces, but we can treat it as one entity here. Our request gets sent to the platform, and given availability, it will grant access to a temporary DNAnexus Worker.

• DNAnexus Worker: This temporary worker is the third key player and is where we do our computation on. We'll see that it starts out as a blank slate.

Specific Terms Outside of Key Players

• A project contains files and executables and logs associated with analysis securely stored on the platform

• The executables on the platform are referred to as apps. Apps are executables that can be run on the DNAnexus platform. Most importantly, they need to contain a software environment to run the executable.

• A software environment in general is everything needed to run software on a brand new computer. This includes the software itself that you are needing as well as any dependencies that are needed to run the software. Some examples of dependencies are languages (such as R) that are needed to execute the software.

Project Storage vs Workers

Project storage is permanent, but the workers are temporary. This means that you have to relay information back and forth as shown in the figure below.

The key concept with cloud computing: project storage can be considered as permanent on the platform. Note that workers are temporary. Because workers are temporary, we need to transfer the files we want to process to them. When we are done, we need to transfer any output files back to the project storage. If we don't do this, the files will be lost when we lose access to the worker.

Local vs Cloud Analysis

Local Machines

• On your local computer, everything is on your machine.

  • This includes your data and the scripts, as well as your software environment and dependencies are also downloaded.

  • The results and in between steps are also generated and saved on your machine as well.

• You own it and you control it.

Cloud Computing

• In comparison, cloud computing adds layers into analysis to increase computational power and storage.

• This relationship and the layers involved are in the figure below:
• Let's contrast this with the process of processing a file on the DNAnexus platform.

Key Differences

• The first difference is that we need to request a worker and we only have temporary access to it. We need to bring everything to the worker, including the software environment.

• The second key difference is that we need to bring our files and scripts from project storage to the worker.

Common Challenges with Cloud Computing

Challenge 1: Requesting Enough Resources

• Our first barrier is requesting an appropriate worker that can do our computational job.

• For example, our app may require more memory, or if it is optimized for working on multiple CPUs, more CPUs.

• We need to understand how big our files are and the computing requirements of our software to do this.

Challenge 2: Installing Dependencies

• Our second barrier is installing the software environment on the worker, such as R.

• Because we are starting from scratch on a worker, we will need ways to reproducibly install the software environment on the worker.

• We'll see that this is one of the roles of Apps. As part of their job, they will install the appropriate software environment.

Resolution for Challenge 1 and 2:

• There is some good news. If we are running apps, they will handle both of these barriers.

• Number one, all apps have a default instance type to use. We'll see that we can tailor this.

• Secondly, Apps install the required software environment on their workers.

Challenge 3: Transferring Files

• Our third barrier is getting our files onto the worker from project storage, and then doing computations with them on the worker. The last barrier we'll talk about is getting the file outputs we've generated from the worker back into the project storage.

• Cloud computing has a nestedness to it and transferring files back and forth can make learning it difficult.

• Having a mental model of how cloud computing works can help us overcome these barriers.

Resolution for Challenge 3:

• Cloud computing is indirect, and you need to think 2 steps ahead.

• Here is the visual for thinking about the steps for file management:

Solution for Challenges: Apps

• Apps help you address installing software on worker

• Prebuilt software environment that is installed onto the temporary worker

• Can build our own apps

• Apps serve to (at minimum):

Resources

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform

2. Select "Contact Support"

3. Fill in the Subject and Message to submit a support ticket.

To filter with gene expression data, you can add a filter based on the tiles created in the Gene Expression tab or use the “+ Add Filter” button in the Cohort Banner.

These filters allow for:

• Assessing impact of genes/ features and their expression levels

• Building Cohorts based on Gene Expression Level

You Can Filter By:

• Gene Symbol or Ensembl ID with Expression Level

Adding Gene Expression Filters

1. Add in your dataset

2. Select "+ Add Filter"

1. Select Assays and then under Gene Expression, select “Features/ Expression”

1. Select the genes that you want as well as the expression range. Please note, for the Gene/ Feature value, you can select by Gene Symbol or the ENSEMBL ID.

Supplemental Video for Gene Expression Tab and Filtering

Please note: in order to use Cohort Browser on the Platform, an Apollo License is needed.

A Tour of the Cohort Browser

Purpose of the Cohort Browser

The cohort browser is used for browsing and visualizing data and creating cohorts. These cohorts can then be shared in a project space to your collaborators.

Metadata keeps your data objects and projects organized. All objects that are uploaded or created will have associated metadata with fields such as Name, ID, Path, Status, Class, File Size, Created by, Created, and Modified.

Data Objects

Object Classes include:

• Data files

Projects have a series of features designed to facilitate collaboration, help project members coordinate and organize their work, and ensure appropriate control over both data and tools.

Creating a Project

All work takes place in the context of a project. Projects allow a defined set of users and orgs to:

• Access specific data

What is a Workflow?

• The individual apps can be easily combined into pipelines, which are on the DNAnexus platform referenced as workflows.

• These apps are linked together by dependencies and can hand off their outputs to other apps as they complete.

Please note, the data present is intended for training purposes only. Information about the data present in this documentation is listed .

When somatic variant data is present in your data ingestion for the cohort, the Somatic Variants tab will appear in the Cohort Browser. The goal of viewing data within the Somatic Variants tab is to view somatic mutations present in your data, and to explore variants and events for certain genomic regions. You can also compare these values within 2 different cohorts, as long as they have the same underlying database.

Features of the Somatic Variants Tab

In order for Nextflow to run correctly on the platform, please do the following:

1. Install dxpy/ dx-toolkit. Details on how to do this is in the Command Line Interface Section under Introduction to the CLI.

   1. As Nextflow on DNAnexus is being updated with bugfixes and improvements on a regular basis, we recommend updating dxpy to the latest version prior to building your Nextflow applet.

   2. You can upgrade dxpy by using the following

To filter with somatic data, use the “+ Add Filter” button in the Cohort Banner.

These filters allow for:

• Assessing impact of ingested somatic variants in cohorts

Since the JupyterLab jobs are hosted on a temporary worker, you will either need to download the software packages every time you start a job, or to save a snapshot of the software environment.

What is a Snapshot?

• A snapshot saves the current software environment in the JupyterLab environment.

DX Notebooks Naming

All notebooks saved onto the platform will have a DX prefix in front of them. Here is an example:

To upload your files, you will need to do the following:

• create a folder with the org name for the portal. It will be org-NAME OF COMMUNITY

• Make sure all of your json files are in the folder

• Make sure all of your assets/ images are in the folder.

In this section, we will build several native bash applets that will increase in complexity:

1. An applet that takes an input files, runs a single Unix command, and returns the result as a file.

2. An applet that includes a binary executable file in the resources directory.

3. An applet that installs the dependency cnvkit

Please note, the data present is intended for training purposes only. Information about the data present in this documentation is listed here.

When gene expression data is present in your data ingestion for the cohort, the Gene Expression tab will appear in the Cohort Browser. The goal of viewing data within the Gene Expression tab is to view gene expression values in your cohort, and to compare between 2 cohorts within the same data base.

Features of the Gene Expression Tab

Overview of the Gene Expression Tab

Within the Gene Expression Tab, there are the following sections: plots for Gene Expression, where you can search for genes by Gene Symbol or Ensembl ID, and an Expression per Feature table. The tables and figures of the Gene Expression Tab are highlighted in the figure below:

Gene Expression Plots

To view gene expression for a specific gene, type the gene symbol or Ensembl ID into the search bar for the charts labelled “Expression Level”. There are 3 options for the plots: Expression Level with a box plot, Expression Level with a histogram, and a Feature Correlation scatter plot between 2 genes. More than 3 tiles can be added with the “Add Tile” button, and typing in the Gene Symbol or Ensembl ID.

Expression Level Box Chart

For the box plot, you can see the distribution of the expression level for a given gene by typing in the gene symbol or Ensembl ID to the search bar. You can view the detailed distribution as a violin plot, or as a box plot. The x axis is the distribution of gene expression levels in the cohort and the y axis is the Expression Level. The options to view the detailed distribution are part of the Chart Settings. The Bar Chart with the Violin Plot (detailed distribution) is shown below:

Expression Level Histogram

For the histogram, you can see the distribution of the expression level for a given gene by typing in the gene symbol or Ensembl ID to the search bar. You can see the histogram with or without the display statistics. The x axis is the distribution of gene expression levels in the cohort and the y axis is the Expression Level. The options to view the detailed distribution are part of the Chart Settings. The histogram with the display statistics settings are shown below:

Feature Correlation

For the feature correlation, you can see the expression level for a given gene for the x and y axis by typing in the gene symbol or Ensembl ID to the search bar. You can see the feature correlation with or without the display statistics. The x axis is the gene expression level for one gene and the y axis is the gene expression level for another gene. The options to view the detailed distribution are part of the Chart Settings. The feature correlation with the display statistics settings are shown below:

Building Applets

DNAnexus apps and applets are ways to package executable code. The biggest difference between apps and applets is their visibility. Apps such as you find in the Tool Library are globally available and maintained by DNAnexus and partners like Nvidia and Sentieon. Applets are private to an organization and exist as data objects in a project. They can be shared across projects and promoted to generally available apps. Native DNAnexus applets are built using dx build to create an executable for bash or Python code, which in turn may execute any program installed on the instance.

Later, we will discuss how to build a workflow, which is a combination of two or more apps/applets. We will build native workflows using the GUI and languages like WDL (Workflow Description Language) and Nextflow combine with Docker images.

Development Cycle

As shown in following figure, the development cycle is to write code locally, use dx build to create a native applet on the platform, and then dx run to run the applet. You can view the execution logs with dx watch, then make changes to your code to build and run again.

Installing Required Software

To install the Python modules required for this tutorial, run the following command:

You may be prompted to expand PATH with installation directory such as ~/.local/bin:

Next, ensure you have a recent version of Java. For this tutorial, I'm using the following:

If you want to use to execute WDL locally, you should download the Cromwell Jar file. This tutorial assumes you will place this file in your home directory using the following commands.

I suggest you use the link command (ln) to create a symlink to the filename cromwell.jar so that upgrading in the future will not break your commands:

(Workflow Object Model) is also quite useful, and I suggest you similarly download it and link it to womtool.jar:

You will use the DNAnexus to build WDL applications on the platform. Find a link to the latest Jar file under the releases of the . For example, the following commands will download dxCompiler-2.10.4.jar to your home directory and symlink it to dxCompiler.jar:

Some tools may attempt to use the tool to validate any shell code in your WDL. To install on Ubuntu, run the following:

On macOS, you can use to install the program:

The dx CLI

If the dxpy module installed properly, you should able to run dx on the command line. For instance, run dx all to see :

To get started, do the following:

• Run to identify yourself to the DNAnexus platform. Enter your username and password. You can also set up a token to log in. Information on setting up tokens can be found in the section of our Documentation.

• You may also be prompted to select a project. If not, you should use to select a project that will contain your work.

• If you do not see a project you wish to use for your work, run dx new project to create one from the command line, or click "New Project" in the web interface.

Note that each subcommand will respond to the flags -h|--help to display the usage documentation. For instance, dx new can create several object types, which you can discover by reading the documentation:

You should now be prepared to develop DNAnexus apps and workflows.

Resources

Please email to create a support ticket if there are technical issues.

To filter with gene expression data, you can add a filter based on the tiles created in the Gene Expression tab or use the “+ Add Filter” button in the Cohort Banner.

These filters allow for:

• Creating a more complex cohort with Phenotypic and genomic filtering

You Can Filter By Combinations with the following data:

• Phenotype/ Clinical data

• Germline Variants

• Somatic Variants

• Gene Expression Changes

Adding Multiple Filters

1. Add in your dataset

2. Select "Add Filter"

1. Choose the filtering that you are interested in. Details for Phenotype Filtering, Germline Filtering, Somatic Filtering, and Gene Expression are available in previous sections of the documentation. (These will have links).

2. Once the initial filter is complete, select “Add Additional Criteria” next to the filter, as shown below:
3. Repeat the process for the next cohort filter that you need.

Supplemental Videos for the Cohort Browser

Combining Cohort Filters: Phentotypic and Germline Variants

Combining Cohort Filters: Phenotypic and Somatic Variants

Combining Cohort Filters: Phenotypic and Gene Expression

Starting a TTYD Instance

You can start a TTYD job the same way as you would any other job in the UI.

1. Select Start Analysis in the top right corner in the project space.

1. Select the app called “ttyd”.

1. Select Next and then Start Analysis.

1. As the last step before launching the tool, you can review and confirm various runtime settings. Click on Launch Analysis. The job will be launched and you will be redirected to the Monitor tab in a few seconds.

2. In the Monitor tab, select the name of the ttyd job to view more details.

1. Once the state of the job switches to “Running”, you will be able to enter the ttyd with the “Open Worker URL” link in the top heading of the details page. If the page to which you get redirected says “502 Bad Gateway”, the worker is not yet fully initialized. Close the page, give it a few more minutes and try to open the worker URL again.

1. This will open a terminal in your browser that will give you access to the files in the DNAnexus project in which the app is running by mounting it in a read-only mode in the /mnt/project directory of the worker execution environment.

2. Once you are done with your work in ttyd, don’t forget to terminate the job by clicking the red button Terminate in the job’s details page.

Resources

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform

2. Select “Contact Support”

3. Fill in the Subject and Message to submit a support ticket.

TTYD vs Cloud Workstation

Resources

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform

2. Select “Contact Support”

3. Fill in the Subject and Message to submit a support ticket.

To filter with germline data, use the “+ Add Filter” button in the Cohort Banner.

These filters allow for:

• Assessing impact of ingested variants in cohorts

• Note: only non-ref variants are represented in the genomic data

• Building Cohorts based on Variants

• Develop basket studies based on your population

• Exploratory Data Analysis before GWAS

• Ask questions about co-occurrence with other mutations

You Can Filter By:

• Gene Symbol or region

• Variant effect

• Variant type

• Variant ID

Adding Germline Filters

1. Add in your dataset

2. Select "+ Add Filter"

1. Select Assays and then under Genome Sequencing, select “Genes/ Effects”

1. Then, select the genes/ impact/ variants that you want. Please note, the filtering for the Genes/ Genomic regions is by Gene Symbol or Genomic range.

Supplemental Video: Germline Data Tab + Filtering

Disclaimer: Portals require a license. These documents are to get you started with your portals. By no means is this the only way to make your portal, nor is this the only way to edit a json file.

Each section of a portal has a different json file.

Here is a visual of which json file edits which section of a portal:

navigation.json

This section defines the following:

• navigation/ header bar

• items that are in the header after the logo that are also not included in the branding.json

• You can also add/ delete navigation items

branding.json

This section defines the following:

• logo

• colors

• if you want a login page

• a home URL attached to the logo

home.json

• This controls the home page for the community portal

• You can specify the following:

  • order of the sections

  • components

Resources

Please email to create a support ticket if there are technical issues.

Resources for JSON

• What are JSON files and how do you use them?

• What is a JSON file?

JSON File Basics

• JavaScript Object Notation

• Common format for communicating with Application Program Interface (API)

• Used to access DNAnexus API servers

Why Should You Learn JSON?

• Reading and modifying JSON is at the heart of building and running apps

• Understanding JSON responses from the API will help you debug jobs

• Automation and Batch submissions: running the same app on multiple files

  • Find which jobs have failed and why

Elements and Reading JSON

A valid JSON document is enclosed in one of two data structures, either a list of values contained in square brackets:

Or an object composed of key/value pairs contained in curly brackets:

Example:

A JSON value may be any of the following:

• single- or double-quoted string, e.g., "samtools" or 'file-G4x7GX80VBzQy64k4jzgjqgY'

• integer, e.g. 19 or -4

• float, e.g., 3.14 or 6.67384e-11

Lists

• Lists are braced in square brackets [ ]

• Similar to Python syntax

• Used for multiple values separated by commas

Example:

Objects

• An object starts and ends with curly braces

• An object contains key/value pairs

• Keys must be quoted strings

• Values may be any JSON value, including another object

Example:

Resources

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform

2. Select "Contact Support"

3. Fill in the Subject and Message to submit a support ticket.

You also do not need to define executor like you might do for some other cloud Nextflow executors. By default the executor is 'local'. However, if you are for instance going to be running Nextflow in multiple locations and want different settings based on location you could set a DNAnexus profile in your nextflow.config which explicitly defines the executor and things like default queueSize.

Here is an example DNAnexus executor profile which also enables docker.

when running on DNAnexus you would then give '-profile dnanexus' to 'nextflow_run_opts' in the UI or in the CLI it would be -inextflow_run_opts='-profile dnanexus'

You could also create a test profile for testing on your own servers/cloud workstation and on DNAnexus.

Tips and Tricks

• If pipeline contains inputs from external sources (such as S3, FTP, HTTPS), those files are staged in the head-node and may run out of storage space (inputs sources from DNAnexus are not staged in this way).

  • The instance size of the head-node can be customized: in "Applet Settings" on the UI with the --instance-type flag on the CLI

• 20 sessions can be cached per project

  • The number of times any of those sessions can be resumed is unlimited

Resources

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform

2. Select "Contact Support"

3. Fill in the Subject and Message to submit a support ticket.

Some of the links on these pages will take the user to pages that are maintained by third parties. The accuracy and IP rights of the information on these third party is the responsibility of these third parties.

You can collaborate on the DNAnexus Platform by giving project access to other users. Project access can be revoked at any time by a project administrator.

Adding Project Members

Once you've created a project, you can add members by doing the following:

1. From the project's Manage screen, click the Share Project button - the "two people" icon - in the top right corner of the project page.

Use Cases for Spark JupyterLab Instances

• Utilization of load_cohort. It requires running SQL on Spark and has a specialized functionality that we only support via dxdata (python).

• Complex interactions with records/Spark must be done via Python.

Here is the overview of

To use an gatk image (made in previous sections) with the Swiss Army Knife tool, you will do the following:

Command Line

From there, this will be what the command line prompts will be:

From there, you will see the job log of the Swiss Army Knife App.

Nextflow has resources depending on where your skill level is with using their workflow language.

The general page for these resources can be found at and

Nexflow also has a general and

New to Nextflow Users

This is a walk through of how to add an existing Docker Image to the platform and saving it as a snapshot file on the platform

To get started with this, you will either need to 1) open a ttyd or 2) have Docker installed and use you local terminal with dxtoolkit installed as well.

Overview of How to Use the Docker Image to Snapshot File

Why Docker?

• Portability - Run code and scripts on any machine that has Docker installed

  • Avoids installation headaches on multiple machines (dependencies are installed with software)

Disclaimer: Portals require a license. These documents are to get you started with your portals. By no means is this the only way to make your portal, nor is this the only way to edit a json file.

Overview of the navigation.json file

• this .json file will personalize the banner that you navigate to different sections.

If you also have access to the ML JupyterLab (another solution in the AI/ML Accelerator Package), Data Profiler can be seamlessly opened in the JupyterLab environment, offering an intuitive and interactive tool for profiling multiple datasets directly within one workspace.

To get started, simply open an ML JupyterLab notebook, load the dataset, and profile it.

Profiling the Dataset

The integrated version of Data Profiler in ML JupyterLab (dxprofiler) offers four methods for loading your datasets to profile the data:

Account Management

Glossary of Terms

Within a project space, there are different permission/access levels. They are:

Defining Members of an Org and their Relationship

Why Add Users to an Org?

• Allows the access to the shared apps

  • This is for what the org is an authorized user for

  • If the org cannot use the app, the member cannot either

• Allow seeing the price column in the UI monitor tab and on the command line

Project Overrides for a User

• By default, when a project is created, the settings tab shows the following:
• The owner of the project can change these

• You may want to restrict them depending on your org policy

• Copy access

Sharing in an Org

The org allows for the sharing

• of the same resources

  • Control the access as stated above

  • Org admins can remove and add users

• to users performing similar functions

Example 1: Orgs and Sharing

• Sharing projects and apps within orgs allows a group of users performing similar functions to be given the same level of access to shared resources.

• In this example, there is the org administrator, admin a, who provides view access to the project resources to the org. Additionally, admin A adds users B and C to the org, and also adds admin D to the org.

• Admin D then provides upload permissions to the project raw data, and makes the org and authorized user of the QC app. So in addition to being a convenient way to share projects and data, the org aids provide access to apps as well.

Multiple Orgs

You can have multiple people in multiple orgs.

Example 2: Multiple Orgs

• Members who are working on two separate projects, and they need access to different data/ apps which have different budgets.

• the user may need to create and work on projects that are billed to two step or groups in. This is where creating multiple orgs comes in handy.

• Admin D is admin of both org and org-new because admin D needs to work within both of these orgs.

• Admin D adds user E to both org and org- new and only adds member or user F to org- new because user only needs to work within org-new.

Resources

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform

2. Select "Contact Support"

3. Fill in the Subject and Message to submit a support ticket.

Before you begin, review the overview documentation and log onto the DNAnexus Platform

Tool Library

The tool library is a set of ready to use apps and work flows that are maintained by DNAnexus

There are different categories and you can search by name of the tool.

Steps to finding Tool Documentation

1. Navigate to the Tool Library

2. In the Any Name search box, start entering "FASTQC...."

3. Click on the tool name, and you will be at the info tab of the tool.

4. Select the Version: If you want the same version that is loaded automatically, this is all that you will need to do. If you want a different Version, select the Versions tab and select which version you want.

You can also select "Run" to run the app

Tool Runner Options

There are 2 options for running the tool. First, select "Run" where you find the tool documentation.

Then, there are 2 different UIs for setting up the app to run:

The guided set up, which is what you normally start with

Or, the I/O graph

Tool Useful Features

In the Stage settings tab, you can set the version of the app you want to use, instance type and specify the output folder. By specifying the instance type, you will set the computational resources of the machine on which the analysis will be run. For example, if your input data is large, you will choose an instance type with more storage space available.

Required inputs indicated by asterisks, some are optional.

It is point and click.

Can select your instance here.

can be enabled here. At this time, the feature applies to a batch of inputs. The output is aggregated in one output file. (e.g. 10 inputs results in 1 output).

Running and Monitoring an App

Set Up

• Once you have selected the app you want to use and read the documentation (if applicable), you will use the guided setup to run the app in the UI.

• Set the Output folder

• Set the inputs. In the example of FASTQC, it is one FASTQ file

• Launch the app using the start analysis button in the upper right

Monitor An App

• You will automatically be redirected to the monitor page

• When the job is completed, you will have buttons to access the inputs (such as a FASTQ file) and outputs (such as an HTML file).

• Here is the view when the app is completed:

Supplemental Information

Using Apps in the GUI

Batch Processing in the GUI

Monitoring An App/ Workflow

Resources

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform

2. Select "Contact Support"

3. Fill in the Subject and Message to submit a support ticket.

New to JupyterLab?

If you have never used a JupyterLab notebook before, please view this information:

• Jupyter Notebook Documentation

Introduction

We can interact with the platform in several different ways and install software packages in these different environments depending on what we are wanting to use and how we want to use it. As shown in the diagram below, we will be explaining Jupyter Lab Python/R/Stata and Spark JupyterLab Python/R:

Why JupyterLab?

Data Scientists’ tasks can be interactive. Options for interactive analysis in JupyterLab are:

• Notebook-based Analysis

• Exploratory Data Analysis (EDA)

• Data Preprocessing/ Cleaning

• Implementing New Machine Learning(ML)/ Model

Requesting an Instance

Use Single DXJupyter Instance if:

• The work can be done on a single machine instance

• Main Use Cases:

• Python/R

• Image Processing

Use Spark Cluster DXJupyter If:

• Working with very large datasets that will not fit in memory on a single instance

• Using the Cohort Browser and querying a large ingested dataset

• Needing to use Spark based tools such as dxdata, HAIL or GLOW

Starting a JupyterLab Job

1. Select JupyterLab with Python, R, Stata, ML, Image Processing or JupyterLab from Spark from the Tool Library, or select “Start Analysis” from the project space and select JupyterLab from the tool list. Once selected, press “Run Selected”

1. Select the output location, and change the job name if desired.

1. Then, select the inputs you intend on using

   1. Snapshot file (not required, and how to create a snapshot is in the Utilizing Snapshot section)

   2. Input files (not required, can do in the notebook analysis)

   3. Stata settings file (license required for Stata)

1. Then, press “Start Analysis” in the far right corner

1. Next, confirm the following parameters:

   1. Job Name

   2. Output Folder

   3. Priority (defaults to normal, can be set to high)

1. Then, press “Launch Analysis”

2. When redirected to the monitor tab, select the job name

3. It will redirect you to the details of the JupyterLab job. Wait for the job to start running, and for the worker URL to appear

4. Press “Open Worker URL” and the JupyterLab home page will appear

1. Note: Sometimes, the job is still initializing, so if you press Open Worker URL immediately, it may show a 502 error message. This is okay, and the job will update when the job is finished initializing.

Running instances may take several minutes to load as the allocations become available.

Nextflow's errorStrategy directive allows you to define how the error condition is managed by the Nextflow executor at the process level.

There are 4 possible strategies:

The DNANexus nextflow documention has a

Generally the errorStrategy is defined in either the base.config (which is referenced using includeConfig in the nextflow.config file) or in the nextflow.config file.

In nfcore pipelines, the default errorStrategy is usually defined in base.config and it is set to 'finish' except for error codes in a specific numeric range which are retried.

The code below is from the

The maxRetries directive allows you to define the maximum number of times the exact same subjob can be re-submitted in case of failure and the maxErrors directive allows you to specify the maximum number of times a process (across all subjobs of that process executed) can fail when using the retry error strategy. .

In the code above, if the exit status of the subjob (task) is within 130 to 145, inclusive, or is equal to 104, then it will retry that subjob once (maxRetries = 1). If other subjobs of the same process also have the same issue, they will also be retried once (maxErrors = '-1' disables the max number of times any process can fail so if every subjob executed for a particular process failed it will allow it to be retried the number of times set in maxRetries). Otherwise, the finish errorStategy is applied and the subjob is terminated pending but other running non-errored subjobs are allowed to complete.

For example, imagine you have a fastqc process that takes in one file at a time from a channel with 3 files (file_A, file_B, file_C)

The process is as below and is run for each file in parallel

• fastqc(file_A)

• fastqc(file_B)

• fastqc(file_C)

If the subjob with file_A and the subjob with file_C fail first with errors in range 130-145 or with a 104 error, they can each be retried once if maxRetries =1 .

Now imagine that you set maxErrors = 2. In this case, there are 3 instances of the process but only 2 errors are allowed for all instances of the process. Thus, it will only retry 2 of the subjobs e.g. fastqc(file_A) fastqc(file_C)

If fastqc(file_B) encounters an error at any point, it won't be retried and then the whole job will go to the finish errorStrategy.

Thus, disabling the maxErrors directive by setting it to '-1' allows all failing subjobs with the specified error codes to be retried X amount of times with X set by maxRetries.

Debugging Checklist for Errors

• Check what version of dxpy was used to build the Nextflow pipeline and make sure it is the newest

• Look at head-node log (hopefully it was ran with "debug mode" as false because when true, the log gets injected with details which isn't always useful and can make it hard to find errors)

  • Look for the process (sub-job) which caused the error, there will be a record of the error log from that process, though it may be truncated

Resources

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform

2. Select "Contact Support"

3. Fill in the Subject and Message to submit a support ticket.

Some of the links on these pages will take the user to pages that are maintained by third parties. The accuracy and IP rights of the information on these third party is the responsibility of these third parties.

Please note: in order to use Cohort Browser on the Platform, an Apollo License is needed.

Cohort Combine Logic

Cohort combine logic allows you to combine existing cohorts with Boolean Logic operations

Overview

Here is a summary of the functions for Cohort Combine

Rules and Boundaries

• All cohorts must be from the same dataset.

• All cohorts must be saved before being combined.

• A cohort that is a result of combine cannot be combined a second time.

• Cohorts from different projects can be combined if they use the same underlying database.

How To:

1. Add your cohorts into the cohort browser by selecting “Load Saved Cohort” if the cohort has already been created and saved into the project, or “New Cohort” if a new cohort needs to be created. You can select up to 10 cohorts to load into the side menu.

1. Pick your cohort and add it to the browser. It will look like this.

1. At the bottom of the cohort tab, select "Combine Cohorts"

1. You will then have the following screen to combine. Pick your cohort combine logic, then select combine

Examples of Cohort Combine Functions

Intersection

Overview:

Example:

Union

Overview:

Example:

Subtraction

Important note: the order of the cohorts matters in this.

Overview:

Example 1:

Example 2:

Unique

Overview:

Example:

Complement

Important notes:

• Cohort must be saved before creating its complement (same rule as previous)

• A combined cohort (Intersection, Union, Subtraction, Unique) can be used to create a complement.

• A cohort created as a complement cannot be further used for combine / complement.

Overview:

Example:

Why would you transition from an applet to an app?

Differences between an App and Applet

Checklist of items to keep in mind:

When publishing an app, the following items are needed:

1. A working applet that you have tested

2. A name that is unique. Generally, the recommendation is to have an abbreviation for your org as part of the name. Example: If the org is named “academy_demos” and the app is for fastqc, then the name of the app could be “academy-fastqc”, “academy_demo-fastqc”, or “academydemo-fastqc”.

3. Documentation to add to a README.md for users to understand what your app does

4. Developer notes for you to keep track of version information and added to the Developer README.md

To Publish the App

1. Use dx get applet-name to have the most recent version of your applet

2. Make your changes to the dxapp.json

3. Then use dx build app_name --publish --app

Helpful Trick

• Forget to add users or need to add more users? Use:

Resources

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform

2. Select “Contact Support”

3. Fill in the Subject and Message to submit a support ticket.

Before you begin, set up a DNAnexus Platform account here: https://platform.dnanexus.com/login

Basic Structure

The Platform

There are several ways to interact with the platform. All of these will be covered in future lessons/ courses/ documentation.

We are going to be focused on the user interface (highlighted in green), also known as UI.

Projects

This information can also be found in the for the Platform

First, what is a project?

• It is a collaborative workspace

• The smallest unit of sharing on the platform

• A place to store objects that are made on the platform

  • Examples of these objects can be files, applets, and workflows

Folder Usage

• The user folder is the storage area for your output files

• You can add more folders into your user folder for organization (maybe one for data, each project, etc. This is however you and your organization/ company wants to do this)

Status of a Data Object

Data can be in one of 3 states

• Open: initial, empty state, awaits upload

• Closing: uploading, not instantaneous

• Closed: Finalization completed, available for next steps

Creating Folders

1. Log into the

2. When you login, you will see a list of projects that you are a part of.

3. Navigating to a project

   1. We have prebuilt projects for you

Copying Files

• Copying means from one project to another project

• You cannot copy within the same project because of the file ID.

Resources

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform

2. Select "Contact Support"

3. Fill in the Subject and Message to submit a support ticket.

Use Cases for a Single JupyterLab Instance

• R needs to run in a regular notebook and for downstream analysis

• If directly interacting with the database/ dataset, it is recommended that you either 1) use Python and/ or 2) use Spark for extracting the data that is relevant for the downstream analysis

General “Recipe” for Utilizing Single Instance JupyterLab Notebooks

1. Create a DX JupyterLab Notebook so that it will automatically save onto the Trusted Research Environment. You can do so by selecting these 2 different options:

   1. Option 1 is from the Launcher:

   b. Option 2 is from the DNAnexus Tab:

1. Start writing your JupyterLab Notebook. Select which kernel you are going to use (options will vary depending on the Image you selected in set up).

2. Download packages and save the software environment as a snapshot

   1. Download Packages

   b. Save the Snapshot of the environment

Opening Notebooks from Project Storage

• Notebooks can also be directly opened from project storage

• When you save in JupyterLab, the notebook gets uploaded to the platform as a new file. This goes back to the concept of immutability.

• The old version of notebook goes into .Notebook_archive/ folder in project.

A license is required to access the Data Profiler on the DNAnexus Platform. For more information, please contact DNAnexus Sales (via [email protected]).

A Note on Data:

The data used in this section of Academy documentation can be found here to download: https://synthea.mitre.org/downloads

The citation for this synthetic dataset is:

Walonoski J, Klaus S, Granger E, Hall D, Gregorowicz A, Neyarapally G, Watson A, Eastman J. Synthea™ Novel coronavirus (COVID-19) model and synthetic data set. Intelligence-Based Medicine. 2020 Nov;1:100007. https://doi.org/10.1016/j.ibmed.2020.100007

PygWalker and Explorer Mode

PygWalker shows a sample of the dataset in a table format

PygWalker simplifies data analysis and visualization by transforming pandas dataframes into an interactive interface for easy exploration. It is available within the table-level view of the application. To use it, simply click the Go to Explorer Mode button to access the raw data slide. You can learn more about its features by referring to the documentation or watching demo videos .*

A custom plot created with PygWalker

*DNAnexus is not responsible for the accuracy or updating of any 3rd party content or applications*

Resources

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform

2. Select "Contact Support"

3. Fill in the Subject and Message to submit a support ticket.

If your nextflow run fails, the nextflow job log is written to your project Output location (CLI flag --destination) that you set for the applet at runtime.

However, on failure, your results files in params.outdir are not written to the project, unless you are using the 'ignore' error strategy.

To guard against having long running or expensive (or both!) runs that you get no output from when they fail you need to think carefully about what should happen when your job fails and if you need the ability to resume it. This means that successfully completed processes won't be run again saving you the cost and time of running already successfully completed jobs.

Nextflow has a resume feature to enable runs that fail to be resumed again which

Caching the Nextflow workDir

A license is required to access the Data Profiler on the DNAnexus Platform. For more information, please contact DNAnexus Sales (via ).

A Note on Data:

The data used in this section of Academy documentation can be found here to download:

The citation for this synthetic dataset is:

Walonoski J, Klaus S, Granger E, Hall D, Gregorowicz A, Neyarapally G, Watson A, Eastman J. Synthea™ Novel coronavirus (COVID-19) model and synthetic data set. Intelligence-Based Medicine. 2020 Nov;1:100007.

Disclaimer: Portals require a license. These documents are to get you started with your portals. By no means is this the only way to make your portal, nor is this the only way to edit a json file.

Overview of the branding.json file

• this .json file will edit the branding portion of the portal.

HPC vs the DNAnexus Platform