1 Introduction

A central concept for designing workflows within the systemPipeR environment is the use of workflow management containers. systemPipeR adopted the widely used community standard Common Workflow Language (CWL) (Amstutz et al. 2016) for describing analysis workflows in a generic and reproducible manner. Using this community standard in systemPipeR has many advantages. For instance, the integration of CWL allows running systemPipeR workflows from a single specification instance either entirely from within R, from various command line wrappers (e.g., cwl-runner) or from other languages (, e.g., Bash or Python). systemPipeR includes support for both command line and R/Bioconductor software as well as resources for containerization, parallel evaluations on computer clusters along with the automated generation of interactive analysis reports.

An important feature of systemPipeR's CWL interface is that it provides two options to run command line tools and workflows based on CWL. First, one can run CWL in its native way via an R-based wrapper utility for cwl-runner or cwl-tools (CWL-based approach). Second, one can run workflows using CWL’s command line and workflow instructions from within R (R-based approach). In the latter case the same CWL workflow definition files (e.g. .cwl and .yml) are used but rendered and executed entirely with R functions defined by systemPipeR, and thus use CWL mainly as a command line and workflow definition format rather than software to run workflows. In this regard systemPipeR also provides several convenience functions that are useful for designing and debugging workflows, such as a command line rendering function to retrieve the exact command line strings for each data set and processing step prior to running a command line.

This overview introduces how CWL describes command line tools and how to connect them to create workflows. In addition, we will demonstrate how the workflow can be easily scalable with systemPipeR.

2 CWL command line specifications

CWL command line specifications are written in YAML format.

In CWL, files with the extension .cwl define the parameters of a chosen command line step or workflow, while files with the extension .yml define the input variables of command line steps.

Let’s explore the .cwl file:

dir_path <- system.file("extdata/cwl/example/", package="systemPipeR")
cwl <- yaml::read_yaml(file.path(dir_path, "example.cwl"))
  • The cwlVersion component shows the CWL specification version used by the document.
  • The class component shows this document describes a command line tool. Note that CWL has another class, called Workflow which represents a union of one or more command line tools together.
cwl[1:2]
## $cwlVersion
## [1] "v1.0"
## 
## $class
## [1] "CommandLineTool"
  • baseCommand component provides the name of the software that we desire to execute.
cwl[3]
## $baseCommand
## [1] "echo"
  • The inputs section provides the input information to run the tool. Important components of this section are:
    • id: each input has an id describing the input name;
    • type: describe the type of input value (string, int, long, float, double, File, Directory or Any);
    • inputBinding: It is optional. This component indicates if the input parameter should appear on the command line. If this component is missing when describing an input parameter, it will not appear in the command line but can be used to build the command line.
cwl[4]
## $inputs
## $inputs$message
## $inputs$message$type
## [1] "string"
## 
## $inputs$message$inputBinding
## $inputs$message$inputBinding$position
## [1] 1
## 
## 
## 
## $inputs$SampleName
## $inputs$SampleName$type
## [1] "string"
## 
## 
## $inputs$results_path
## $inputs$results_path$type
## [1] "Directory"
  • The outputs section should provide a list of the expected outputs after running the command line tools. Important components of this section are:
    • id: each input has an id describing the output name;
    • type: describe the type of output value (string, int, long, float, double, File, Directory, Any or stdout);
    • outputBinding: This component defines how to set the outputs values. The glob component will define the name of the output value.
cwl[5]
## $outputs
## $outputs$string
## $outputs$string$type
## [1] "stdout"
  • stdout: component to specify a filename to capture standard output. Note here we are using a syntax that takes advantage of the inputs section, using results_path parameter and also the SampleName to construct the output filename.
cwl[6]
## $stdout
## [1] "$(inputs.results_path.basename)/$(inputs.SampleName).txt"

Next, let’s explore the .yml file, which provide the parameter values for all the components we describe above.

For this simple example, we have three parameters defined:

yaml::read_yaml(file.path(dir_path, "example_single.yml"))
## $message
## [1] "Hello World!"
## 
## $SampleName
## [1] "M1"
## 
## $results_path
## $results_path$class
## [1] "Directory"
## 
## $results_path$path
## [1] "./results"

Note that if we define an input component in the .cwl file, this value needs to be also defined here in the .yml file.

2.1 How to connect CWL description files within systemPipeR

SYSargsList container stores all the information and instructions needed for processing a set of input files with a single or many command-line steps within a workflow (i.e. several components of the software or several independent software tools). The SYSargsList object is created and fully populated with the SYSargsList construct function.

The following imports a .cwl file (here example.cwl) for running the echo Hello World example.

HW <- SYSargsList(wf_file="example.cwl", input_file="example_single.yml", dir_path = dir_path)
HW
## Instance of 'SYSargsList': 
##     WF Steps:
##        1. Step_1 (Status: Pending)
## 
cmdlist(HW)
## $Step_1
## $Step_1$defaultid
## $Step_1$defaultid$example
## [1] "echo Hello World! > results/M1.txt"

However, we are limited to run just one command line or one sample in this example. To scale the command line over many samples, a simple solution offered by systemPipeR is to provide a variable for each of the parameters that we want to run with multiple samples.

Let’s explore the example:

yml <- yaml::read_yaml(file.path(dir_path, "example.yml"))
yml
## $message
## [1] "_STRING_"
## 
## $SampleName
## [1] "_SAMPLE_"
## 
## $results_path
## $results_path$class
## [1] "Directory"
## 
## $results_path$path
## [1] "./results"

For the message and SampleName parameter, we are passing a variable connecting with a third file called targets.

Now, let’s explore the targets file structure:

targetspath <- system.file("extdata/cwl/example/targets_example.txt", package="systemPipeR")
read.delim(targetspath, comment.char = "#")
##                Message SampleName
## 1         Hello World!         M1
## 2           Hello USA!         M2
## 3 Hello Bioconcudctor!         M3

The targets file defines all input files or values and sample ids of an analysis workflow. For this example, we have defined a string message for the echo command line tool, in the first column that will be evaluated, and the second column is the SampleName id for each one of the messages. Any number of additional columns can be added as needed.

Users should note here, the usage of targets files is optional when using systemPipeR's new CWL interface. Since for organizing experimental variables targets files are extremely useful and user-friendly. Thus, we encourage users to keep using them.

2.1.1 How to connect the parameter files and targets file information?

The constructor function creates an SYSargsList S4 class object connecting three input files:

- CWL command line specification file (`wf_file` argument);
- Input variables (`input_file` argument);
- Targets file (`targets` argument).

As demonstrated above, the latter is optional for workflow steps lacking input files. The connection between input variables (here defined by input_file argument) and the targets file are defined under the inputvars argument. A named vector is required, where each element name needs to match with column names in the targets file, and the value must match the names of the .yml variables. This is used to replace the CWL variable and construct all the command-line for that particular step.

The variable pattern _XXXX_ is used to distinguish CWL variables that target columns will replace. This pattern is recommended for consistency and easy identification but not enforced.

The following imports a .cwl file (same example demonstrated above) for running the echo Hello World example. However, now we are connecting the variable defined on the .yml file with the targets file inputs.

HW_mul <- SYSargsList(targets=targetspath, 
                      wf_file="example.cwl", input_file="example.yml", dir_path = dir_path, 
                      inputvars = c(Message = "_STRING_", SampleName = "_SAMPLE_"))
HW_mul
## Instance of 'SYSargsList': 
##     WF Steps:
##        1. Step_1 (Status: Pending)
## 
cmdlist(HW_mul)
## $Step_1
## $Step_1$M1
## $Step_1$M1$example
## [1] "echo Hello World! > results/M1.txt"
## 
## 
## $Step_1$M2
## $Step_1$M2$example
## [1] "echo Hello USA! > results/M2.txt"
## 
## 
## $Step_1$M3
## $Step_1$M3$example
## [1] "echo Hello Bioconcudctor! > results/M3.txt"