2.9.1 Find your app inputs

Once you have copied the public app admin/sbg-public-data/rna-seq-alignment-star/5 into your project, username/api-testing, the app id in your current project is username/api-testing/newcopyofstar. Conversely, you can use another app you already have in your project for this Quickstart.

To draft a new task, you need to specify the following:

• The name of the task
• An optional description
• The app id of the workflow you are executing
• The inputs for your workflow. In this case, the CWL app accepts four parameters: number, min, max, and seed.

You can always check the App details on the visual interface for task input requirements. To find the required inputs with R, you need to get an App object first.

app <- a$app(id = "tengfei/api-testing-2/newcopyofstar")
# get input matrix
app$input_matrix()
app$input_matrix(c("id", "label", "type"))
# get required node only
app$input_matrix(c("id", "label", "type"), required = TRUE)

Conversely, you can load the app from a CWL JSON and convert it into an R object first, as shown below.

f1 <- system.file("extdata/app", "flow_star.json", package = "sevenbridges")
app <- convert_app(f1)
# get input matrix
app$input_matrix()

##                                id                    label    type
## 1                    #sjdbGTFfile              sjdbGTFfile File...
## 2                          #fastq                    fastq File...
## 3               #genomeFastaFiles         genomeFastaFiles    File
## 4 #sjdbGTFtagExonParentTranscript      Exons' parents name  string
## 5       #sjdbGTFtagExonParentGene                Gene name  string
## 6          #winAnchorMultimapNmax         Max loci anchors     int
## 7             #winAnchorDistNbins Max bins between anchors     int
##   required fileTypes
## 1    FALSE      null
## 2     TRUE      null
## 3     TRUE      null
## 4    FALSE      null
## 5    FALSE      null
## 6    FALSE      null
## 7    FALSE      null

app$input_matrix(c("id", "label", "type"))

##                                id                    label    type
## 1                    #sjdbGTFfile              sjdbGTFfile File...
## 2                          #fastq                    fastq File...
## 3               #genomeFastaFiles         genomeFastaFiles    File
## 4 #sjdbGTFtagExonParentTranscript      Exons' parents name  string
## 5       #sjdbGTFtagExonParentGene                Gene name  string
## 6          #winAnchorMultimapNmax         Max loci anchors     int
## 7             #winAnchorDistNbins Max bins between anchors     int

app$input_matrix(c("id", "label", "type"), required = TRUE)

##                  id            label    type
## 2            #fastq            fastq File...
## 3 #genomeFastaFiles genomeFastaFiles    File

Note input_matrix and output_matrix are useful accessor for Tool, Flow, App object as shown below in App example, you can directly call these two function on a JSON file as well.

tool.in = system.file("extdata/app", "tool_unpack_fastq.json", package = "sevenbridges")
flow.in = system.file("extdata/app", "flow_star.json", package = "sevenbridges")
input_matrix(tool.in)
input_matrix(tool.in, required = TRUE)
input_matrix(flow.in)
input_matrix(flow.in, c("id", "type"))
input_matrix(flow.in, required = TRUE)
output_matrix(tool.in)
output_matrix(flow.in)

In the response body, locate the names of the required inputs. Note that task inputs need to match the expected data type and name. In the above example, we see two required fields:

• fastq: This input takes a file array as indicated by “File…”
• genomeFastaFiles: This is a single file as indicated by “File”.

We also want to provide a gene feature file:

• sjdbGTFfile: A single file as indicated by “File”.

You can find a list of possible input types below:

• number, character or integer: you can directly pass these to the input parameter as it is.
• enum type: Pass this value to the input parameter.
• **file:** This input is a file. However, while some inputs accept only single file (File), other inputs takes more than one file (File arrays, FilesList, or ‘File...’ ). This input require you to pass a Files object (for a single file input) or FilesList object (for inputs which accept more than one file) or simply a list in a “Files” object. You can search for your file by id or by name with an exact match (exact = TRUE), as shown in the example below.

2.9.2 Get your input files ready

fastqs <- c("SRR1039508_1.fastq", "SRR1039508_2.fastq")

# get all 2 exact files
fastq_in <- p$file(name= fastqs, exact = TRUE)

# get a single file
fasta_in <- p$file(name = "Homo_sapiens.GRCh38.dna.primary_assembly.fa",
                   exact = TRUE)
# get all single file
gtf_in <- p$file(name = "Homo_sapiens.GRCh38.84.gtf",
                 exact = TRUE)

2.9.3 Create a new draft task

# add new tasks
taskName <- paste0("tengfei_star-alignment ",date())

tsk <- p$task_add(name = taskName,
                  description = "star test",
                  app = "tengfei/api-testing-2/newcopyofstar/0",
                  inputs = list(sjdbGTFfile = gtf_in,
                                fastq = fastq_in,
                                genomeFastaFiles = fasta_in))

Remember the fastq input expects a list of files. You can also do something as follows:

f1 <- p$file(name = "SRR1039508_1.fastq", exact = TRUE)
f2 <- p$file(name = "SRR1039508_2.fastq", exact = TRUE)
# get all 2 exact files
fastq_in <- list(f1, f2)

# or if you know you only have 2 files whose names match SRR924146*.fastq
fastq_in <- p$file(name = "SRR1039508*.fastq", complete = TRUE)

Using complete = TRUE when items is over 100.

2.9.4 Draft a batch task

Now let’s do a batch with 8 files in 4 groups, which is batched by metadata sample_id and library_id. We will assume each file has these two metadata fields entered. Since these files can be evenly grouped into 4, we will have a single parent batch task with 4 child tasks.

fastqs <- c("SRR1039508_1.fastq", "SRR1039508_2.fastq", "SRR1039509_1.fastq",
            "SRR1039509_2.fastq", "SRR1039512_1.fastq", "SRR1039512_2.fastq",
            "SRR1039513_1.fastq", "SRR1039513_2.fastq")

# get all 8 files
fastq_in <- p$file(name= fastqs, exact = TRUE)
# can also try to returned all SRR*.fastq files
# fastq_in <- p$file(name= "SRR*.fastq", complete = TRUE)

tsk <- p$task_add(name = taskName,
                  description = "Batch Star Test",
                  app = "tengfei/api-testing-2/newcopyofstar/0",
                  batch = batch(input = "fastq",
                                criteria = c("metadata.sample_id","metadata.noexist_id")),
                  inputs = list(sjdbGTFfile = gtf_in,
                                fastq = fastqs_in,
                                genomeFastaFiles = fasta_in))

Now you have a draft batch task, please check it out in the visual interface. Your response body should inform you of any errors or warnings.

Note: you can also directly pass file id or file names as characters to inputs list, the package will first guess if the passed strings are file id (24 bit hexdecimal) or names, then convert it to Files or FilesList object. However, as a good practice, we recommend you construct your files object(e.g. p$file(id = , name = )) first, check the value, then pass it to task_add inputs, this is a more safe approach.

3.1.1 Direct authentication

To use direct authentication, users need to specify one of platform or url, with the corresponding token. Examples of direct authentication:

a = Auth(token = "1c0e6e202b544030870ccc147092c250",
         platform = "aws-us")

The above will use the Seven Bridges Platform on AWS (US).

a = Auth(token = "1c0e6e202b544030870ccc147092c257",
         url = "https://gcp-api.sbgenomics.com/v2")

The above will use the specified url as base URL for the API calls. In this example, the url points to the Seven Bridges Platform on Google Cloud Platform (GCP).

a = Auth(token = "2c0e6e202b544030870ccc147092c257")

The above will use the Cancer Genomics Cloud environment, since no platform nor url were explicitly specified (not recommended).

Note: platform and url should not be specified in the same time.

3.1.2 Authentication via system environment variables

The R API client supports reading authentication information stored in system environment variables.

To set the two environment variables in your system, you could use the function sbg_set_env(). For example:

sbg_set_env(url = "https://cgc-api.sbgenomics.com/v2",
            token = "your_token")

See if the environment variables are correctly set:

sbg_get_env("SB_API_ENDPOINT")
## "https://cgc-api.sbgenomics.com/v2"
sbg_get_env("SB_AUTH_TOKEN")
## "your_token"

To create an Auth object:

a <- Auth(from = "env")

To unset the two environment variables:

Sys.unsetenv("SB_API_ENDPOINT")
Sys.unsetenv("SB_AUTH_TOKEN")

3.1.3 Authentication via user configuration file

You can create an ini-like file named credentials under the folder $HOME/.sevenbridges/ and maintain your credentials for multiple accounts across various Seven Bridges environments. An example:

[aws-us-tengfei]
api_endpoint = https://api.sbgenomics.com/v2
auth_token = token_for_this_user

# This is a comment:
# another user on the same platform
[aws-us-yintengfei]
api_endpoint = https://api.sbgenomics.com/v2
auth_token = token_for_this_user

[default]
api_endpoint = https://cgc-api.sbgenomics.com/v2
auth_token = token_for_this_user

[gcp]
api_endpoint = https://gcp-api.sbgenomics.com/v2
auth_token = token_for_this_user

Please make sure to have two fields exactly named as api_endpoint and auth_token under each profile.

To load the default profile (named [default]) from the default user configuration file ($HOME/.sevenbridges/credentials), please use:

a <- Auth(from = "file")

To load the user profile aws-us-tengfei from this configuration file, change the profile_name:

a <- Auth(from = "file", profile_name = "aws-us-tengfei")

To use a user configuration file from other locations (not recommended), please specify the file path using the argument config_file. For example:

a <- Auth(from = "file", config_file = "~/sevenbridges.cfg",
          profile_name = "aws-us-tengfei")

Note: If you edited the credentials file, please use Auth() to re-authenticate.

3.3.1 offset and limit

Every API call accepts two arguments named offset and limit.

• offset defines where the retrieved items started.
• limit defines the quantity of items you want to get.

By default, offset is set to 0 and limit is set to 50. As such, your API request returns the first 100 items when you list items or search for items by name. To search and list all items, use complete = TRUE in your API request.

getOption("sevenbridges")$offset
getOption("sevenbridges")$limit

3.3.2 Search by ID

When searching by ID, your request will return your exact resource as it is unique. As such, you do not have to manually set offset and limit. It is a good practice to find your resources by their ID and pass this ID as an input to your task. You can find a resource’s ID in the final part of the URL on the visual interface or via the API requests to list resources or get a resource’s details.

3.3.3 Search by name

Search by name returns all partial matches unless you specify exact = TRUE. This type of search will only search across current pulled content, so use complete = TRUE if you want to search across everything.

For example, to list all public apps, use visibility argument, but make sure you pass complete = TRUE to it, to show every single things. This arguments generally works for items like “App”, “Project”, “Task”, “File”, etc.

# first, search by id is fast
x = a$app(visibility = "public", id = "admin/sbg-public-data/sbg-ucsc-b37-bed-converter/1")

# show 100 items from public
x =  a$app(visibility = "public")
length(x)  # 100

x = a$app(visibility = "public", complete = TRUE)
length(x)  # 272 by Nov 2016
# this return nothing, because it is not in the first 100 returned names
a$app(visibility = "public", name = "bed converter")
# this return an app, because it pulls *all* app names and did search
a$app(visibility = "public", name = "bed converter", complete = TRUE)

3.3.4 Experiment with Advance Access features

Similar to offset and limit, every API call accepts an argument named advance_access. This argument was first introduced in August 2017 and controls if a special field in the HTTP request header will be sent, which can enable the access to the “Advance Access” features in the Seven Bridges API. Note that the Advance Access features in the API are not officially released yet, therefore the API usages are subject to change, so please use with discretion.

In addition to modifying each API call that uses Advance Access features, the option can be also set globally at the beginning of your API script. This offers a one-button switch for users who wants to experiment with the Advance Access features. The option is disabled by default:

library("sevenbridges")
getOption("sevenbridges")$advance_access

## [1] FALSE

For example, the “folders” API is in Advance Access as of August, 2017. If we try to use the folders API to create a folder with the advance_access option disabled, it will return an error message:

req = api(
    token = "your_token", path = "files",
    method = "POST", body = list(
        "name" = "new-folder", "type" = "folder",
        "project" = "owner/project"))

httr::content(req)$"message"

## [1] "Advance access feature needs X-SBG-Advance-Access: advance header."

To enable the Advance Access features, one can use

opt = getOption("sevenbridges")
opt$advance_access = TRUE
options(sevenbridges = opt)

at the beginning of their scripts. Let’s check if the option has been enabled:

getOption("sevenbridges")$advance_access

## [1] TRUE

Send the API call again:

req = api(
    token = "your_token", path = "files",
    method = "POST", body = list(
        "name" = "new-folder", "type" = "folder",
        "project" = "owner/project"))

The information of the newly created folder will be returned:

httr::content(req)

## $href
## [1] "https://api.sbgenomics.com/v2/files/{folder_id}"
##
## $id
## [1] "folder_id"
##
## $name
## [1] "new-folder"
##
## $project
## [1] "owner/project"
##
## $parent
## [1] "parent_folder_id"
##
## $type
## [1] "folder"
##
## $created_on
## [1] "2017-08-22T12:49:21Z"
##
## $modified_on
## [1] "2017-08-22T12:49:21Z"

3.7.1 For billing

If no billing group id is provided, this call returns a list of paths used to access billing information via the API. If a username is provided, this call lists all your billing groups, including groups that are pending or which have been disabled. If you specify breakdown = TRUE, the call below returns a breakdown of the spending per-project for the billing group specified by billing_group. Information is also displayed for each of the projects a particular billing group is associated with, including task executions, their initiating user, start and end time, and their cost.

# return a BillingList object
(b <- a$billing())
a$billing(id = b$id, breakdown = TRUE)

3.7.2 For invoices

If no Billing Group id provided, This call returns a list of invoices, with information about each, including whether or not the invoice is pending and the billing period it covers. This call returns information about all your available invoices, unless you use the query parameter bg_id to specify the ID of a particular Billing Group, in which case it will return the invoice incurred by that Billing Group only. If an invoice id is provided, this call retrieves information about the specified invoice, including the costs for analysis and storage and the invoice period.

a$invoice()
a$invoice(id = "fake_id")

Note: Currently, invoice is not an object. Instead, it just returns a list.

3.8.1 List all projects

The following call returns a list of all projects of which you are a member. Each project’s project_id and path will be returned.

a$project()

If you want to list the projects owned by and accessible to a particular user, specify the owner argument as follows. Each project’s ID and path will be returned.

a$project(owner = "tengfei")
a$project(owner = "yintengfei")

To get details about project(s), use detail = TRUE, as shown below.

a$project(detail = TRUE)

3.8.2 Partial match project name

For a more friendly interface and convenient search, the sevenbridges package supports partial name matching. The first argument for the following request is name.

# want to return a project called
a$project("hello")

3.8.3 Create a new project

To create a new project, users need to specify the following:

• name (required)
• billing_group_id (required)
• description (optional)
• tags (optional): This has to be a list(). If you are using the API on the CGC environment, you can create a TCGA Controlled Data project by specifying TCGA in tags.
• type (optional): By default, we create a V2, CWL compatible project.

a$project_new("api_testing_tcga", b$id,
              description = "Test for API")

3.8.4 Create a new project with TCGA controlled data on CGC

Follow the directions above, but pass tcga as a value for tags.

a$project_new("controlled_project", b$id,
              description = "Test for API", tags = list("tcga"))

3.8.5 Delete a project

You can delete a single project by making the request to $delete(). Note that the returned object from a$project() sometimes returns list if you use partial matching by name. The $delete() request cannot operate on a list. If you want to operate on a list of object, read more about batch functions in the relevant section below.

# remove it, not run
a$project("api_testing")$delete()
# check
# will delete all projects match the name
delete(a$project("api_testing_donnot_delete_me"))

3.8.6 Update/edit a project

You can update the following information about an existing project:

• name
• description
• billing_group_id

a$project(id = "tengfei/helloworld")
a$project(id = "tengfei/helloworld")$update(name = "Hello World Update",
                                            description = "Update description")

3.8.7 Project member

a$project(id = "tengfei/demo-project")$member()

m <- a$project(id = "tengfei/demo-project")$
       member_add(username = "yintengfei")

m <- a$project(id = "tengfei/demo-project")$
       member(username = "yintengfei")
m$update(copy = TRUE)

== Member ==
username : yintengfei
-- Permission --
read : TRUE
write : FALSE
copy_permission : TRUE
execute : FALSE
admin : FALSE

m$delete()
# confirm
a$project(id = "tengfei/demo-project")$member()

3.8.8 List all files

To list all files belonging to a project, use the following request:

p <- a$project(id = "tengfei/demo-project")
p$file()

3.9.1 List all files

A better way to list files in a project uses the following:

# first 100 files, default offset = 0, limit = 100
p$file()
# list all files
p$file(complete = TRUE)

Alternatively you can get files from the Auth object.

a$file(project = p$id)
a$file(name = "omni", project = p$id, detail = TRUE)

3.9.2 Search and filter file(s)

p = a$project(id = "tengfei/demo")
p$file(metadata = list(sample_id = "Sample1",
                       sample_id = "Sample2",
                       library_id = "EXAMPLE"),
       tag = c("hello", "world"))

# return single object id is "some_file_id"
p$file(id = "some_file_id")
# return a single object named a.fastq
p$file(name = "a.fastq", exact = TRUE)
# public file search using Auth object
a$public_file(name = "ucsc.hg19.fasta.fai", exact = TRUE)
a$public_file(id = "578cf94a507c17681a3117e8")

# get two files
p$file(name = c("test1.fastq", "test2.fastq"), exact = TRUE)
# get two files from public files using shorthand
a$public_file(name = c("ucsc.hg19.fasta.fai", "ucsc.hg19.fasta"),
              exact = TRUE)

# get matchd the pattern for searching first 100 files
p$file(name = c("gz", "fastq"))
# get all matched files from the project
p$file(name = c("gz", "fastq"), complete = TRUE)
# get all files matched ucsc
a$public_file(name = "ucsc.hg19", complete = TRUE)

p = a$project(id = "tengfei/demo")
p$file(metadata = list(sample_id  = "Sample1",
                       sample_id  = "Sample2",
                       library_id = "EXAMPLE"))

p <- a$project(id = "tengfei/demo")
p$file(tag = c("s1", "s2"))

# list all outputs file from a task id
a$task(id = "53020538-6936-422f-80de-02fa65ae4b39")$file()

# OR
p <- a$project(id = "tengfei/demo")
p$file(origin.task = "53020538-6936-422f-80de-02fa65ae4b39")

3.9.3 Copy a file or group of files

This call copies the specified file to a new project. Files retain their metadata when copied, but may be assigned new names in their target project.

Note that Controlled Data files may not be copied to Open Data projects. To make this request, you should have copy permission within the project you are copying from.

Let’s try to copy a file from the Public File repository. The file’s id is “561e1b33e4b0aa6ec48167d7”.

You must provide

• id: file id or list/vector of files ids
• project: project id.
• name (optional):if omitted, use the same name as the original file name

# 1000G_omni2.5.b37.vcf
fid  <- "561e1b33e4b0aa6ec48167d7"
fid2 <- "561e1b33e4b0aa6ec48167d3"
pid  <- a$project("demo")$id
a$copyFile(c(fid, fid2), project = pid)
a$project(id = pid)$file()

NOTE: To copy a group of files, you need the Auth$copyFile() interface. The id of the files in your project will be different from their id in the Public Files repository.

Alternatively, you copy a single file as shown below.

a$project("hello")$file(id = fid)$copyTo(pid)

3.9.4 Delete file(s)

Note: The delete action only works for one file at a time. Be sure your file call returns a single file rather than a file list.

a$project("demo")$file()[[1]]$delete()
## confirm the deletion
a$project("demo")$file()

You can also delete a group of files or FilesList object, just be careful when using this function.

## return 5 files
a$project("demo")$file("phase1")
## delete all of them
delete(a$project("demo")$file("phase1"))
a$project("demo")$file("phase1")

3.9.5 Download files

To get the download information, please use the following:

a$project("demo")$file()[[1]]$download_url()

To download directly from R, use the download call directly from a single File object.

fid <- a$project("demo")$file()[[1]]$id
a$project("demo")$file(id = fid3)$download("~/Downloads/")

I also created a download function for FilesList object to save your time, as follows:

fls <- a$project("demo")$file()
download(fls, "~/Downloads/")

To download all files from a project, use the following:

a$project("demo")$download("~/Downloads")

3.9.6 Upload files via API

Seven Bridges platforms provide a few different methods for data import:

• Import from FTP or HTTP with the web interface
• The GUI uploader
• The file upload API that you can directly call with the sevenbridges package
• The command line uploader which can be involked with the sevenbridges package

Use the API client uploader by calling the project$upload function to upload a file, a file list, or a folder recursively or via a manifest file.

a  <- Auth(username = "tengfei", platform = "cgc")
fl <- system.file("extdata", "sample1.fastq", package = "sevenbridges")

(p <- a$project(id = "tengfei/quickstart"))

# by default load .meta for the file
p$upload(fl, overwrite = TRUE)
# pass metadata
p$upload(fl, overwrite = TRUE, metadata = list(library_id = "testid2", platform = "Illumina x11"))
# rename
p$upload(fl, overwrite = TRUE, name = "sample_new_name.fastq",
         metadata = list(library_id = "new_id"))

dir.ext <- system.file("extdata", package = "sevenbridges")
list.files(dir.ext)
p$upload(dir.ext, overwrite = TRUE)

dir.ext <- system.file("extdata", package = "sevenbridges")
# enable full name
fls <- list.files(dir.ext, recursive = TRUE, full.names = TRUE)
p$upload(fls, overwrite = TRUE)
p$upload("~/Documents/Data/sbgtest/1000G_phase1.snps.high_confidence.b37.vcf")

## upload all fiels and all metadatas
p$upload(manifest_file = "~/manifest.csv")
## verbal = TRUE, print single file level progress
p$upload(manifest_file = "~/manifest.csv", overwrite = TRUE, verbal = TRUE)
## manifest_metadata = FALSE doens't attach any meta
p$upload(manifest_file = "~/manifest.csv", manifest_metadata = FALSE, overwrite = TRUE)
## filter fils first, upload only files that score < 0.5
p$upload(manifest_file = "~/manifest.csv", overwrite = TRUE, subset = score < 0.5)
## attach all meta except "bad_field" and "sample_id"
p$upload(manifest_file = "~/manifest.csv", overwrite = TRUE,
         subset = score < 0.5, select = -c(bad_field, sample_id))

3.9.7 Upload files via command line uploader

As an alternative to API file uploading, we also offer an interface in the sevenbridges package for uploading (possibly large) files via the Java-based command line uploader.

The first step is to download the command line uploader. With get_uploader(), users can easily download the uploader to a local directory for a specified Seven Bridges platform, for example, the Cancer Genomics Cloud:

get_uploader("cgc", "~/Downloads/")

To list the available projects accessible from your account via the uploader, use cli_list_projects(), with your authentication token for that platform and the directory containing the uploader:

cgc_token = "your_cgc_token"
cgc_uploader = "~/Downloads/cgc-uploader/"

cli_list_projects(
    token = cgc_token,
    uploader = cgc_uploader)

The function will return a character vector containing the available projects which you could upload files to, in the form of username/project-name.

Similarly, to list all the file tags in a project via the uploader, use cli_list_tags():

cli_list_tags(
    token = cgc_token,
    uploader = cgc_uploader,
    project = "username/project-name")

Eventually, use cli_upload() to upload local files to a project:

cli_upload(
    token = cgc_token,
    uploader = cgc_uploader,
    file = "~/example.fastq",
    project = "username/project-name"
)

3.9.8 Update a file

You can call the update() function from the Files object. Using this call, the following can be updated:

• The file’s name.
• The file’s metadata. The metadata is in the form of a list. Note that as this call overwrites all the metadata for a given file, please supply a full list of all the metadata for the file. Conversely, you can use the set_meta() function to patch but not overwrite the metadata for a file.

If no parameters were provided, this call will just get the details of that file and update the object itself.

(fl <- a$project(id = "tengfei/demo-project")$file(name = "sample.fastq"))

== File ==
id : 56c7916ae4b03b56a7d7
name : sample.fastq
project : tengfei/demo-project

Show metadata

# show metadata
fl$meta()

Update meta

fl$update(name = "sample.fastq",
          metadata = list(new_item1 = "item1",
                          new_item2 = "item2",
                          file_extension = "fastq"))
# check it out
fl$meta()

3.9.9 Metadata operations

Learn about metadata fields and their values for the relevant environment: [Seven Bridges Platform] [CGC].

Note that the name of a file is not the same as its ID. A file’s ID is a hexadecimal string automatically assigned to a file in a project. The file’s name is a human-readable string. For information, please see the API overview for the Seven Bridges Platform and for the CGC.

Get metadata

To get the metadata for a file, call the meta() function, as shown below.

# meta is pulling the latest information via API
fl$meta()
# field meta data saved the previous saved one
fl$metadata

Set metadata

Files from curated datasets on Seven Bridges environments have a defined set of metadata which is visible on the visual interface of each environment. However, you can also pass additional metadata for each file which is stored with your copy of the file in your project.

To set metadata but not overwrite all previously stored metadata, call the set_meta() function from the Files object. Unlike the update() function for the Files object, the set_meta() function doesn’t overwrite the metadata field. To overwrite the metadata field, use the update() function or pass the optional argument overwrite = TRUE for the set_meta() call.

fl$set_meta(new_item3 = "item3")
fl
# oops it removed rest of the meta
fl$set_meta(new_item4 = "item4", overwrite = TRUE)
fl

Check details of each metadata field

You can use the Metadata() constructor to check the details of each metadata field. Simply call the function (name of the metadata) to display descriptions and enumerated itemts. Please pay attention to the suggested_values field.

# check which schema we have
Metadata()$show(full = TRUE)
# check details for each, play with it
platform()
paired_end()
quality_scale()

As you can see, some metadata fields have suggested values. To construct the metadata, use Metadata() directly to pass the metadata fields and their values directly into the call. The call will perform the validation.

Metadata(platform      = "Affymetrix SNP Array 6.0",
         paired_end    = 1,
         quality_scale = "sanger",
         new_item      = "new test")

3.9.10 Tag file(s)

You can tag your files with keywords or strings to make it easier to identify and organize files. Tags are different from metadata, and are more convenient and visible from the files list on the visual interface.

Add a new tag

To add a new tag, call the obj$add_tag() method for the File object with a single tag string or a list or vector of tags.

Overwrite the tags for a file

To completely overwrite the tags for a file, use the obj$set_tag() method, which contains the default argument overwrite = TRUE.

Let’s use a file called sample.bam and play with its tags.

p  <- a$project(id = "tengfei/s3tutorial")
fl <- p$file("sample.bam", exact = TRUE)
# show tags for single file
fl$tag()
# add new tags
fl$add_tag("new tag")
# equavilent to
fl$set_tag("new tag 2", overwrite = FALSE)
# set tags to overwrite existing
x <- list("this", "is", 1234)
fl$set_tag(x)
# filter by tags
p$file(tag = c("1234", "new"))
p$file(tag = list("1234", "new"))
p$file(tag = "1234")

Below, find convenient methods for FilesList. Let’s add the tag “s1” to a group files matching “Sample1” and add the tag “s2” to a group of files that match “s2”. Then, we will filter by these tags to obtain specific files.

# work on a group of files
# add tag "s2" to a group of files named with "Sample2" in it
fl2 <- p$file("Sample2")
add_tag(fl2, "s2")
# add tag "s2" to a group of files named with "Sample1" in it
fl1 <- p$file("Sample1")
add_tag(fl1, "s1")
# filter by tag s1 or s2
p$file(tag = "s1")
p$file(tag = "s2")
# get files tagged with s2 and 1234
p$file(tag = list("s2", "s1"))

3.10.1 List all apps

This call lists all the apps available to you.

a$app()
# or show details
a$app(detail = TRUE)

Search for an app by name

To search for a specific app by its name, pass a pattern for the name argument or provide a unique id as shown below.

# pattern match
a$app(name = "STAR")
# unique id
aid <- a$app()[[1]]$id
aid
a$app(id = aid)
# get a specific revision from an app
a$app(id = aid, revision = 0)

List all apps in a project

To list all apps belonging to a specific project, use the project argument.

# my favorite, always
a$project("demo")$app()

# or alternatviely
pid <- a$project("demo")$id
a$app(project = pid)

List all public apps

To list all public apps, use the visibility argument.

# show 100 items from public
x <- a$app(visibility = "public")
length(x)
x <- a$app(visibility = "public", complete = TRUE)
length(x)
x <- a$app(project = "tengfei/helloworld", complete = TRUE)
length(x)
a$app(visibility = "public", limit = 5, offset = 150)

Search through all public apps in all locations

To search for a public app cross all locations, make the following call. Note that this may take a bit of time.

a$app("STAR", visibility = "public", complete = TRUE)

3.10.2 Copy an app

This call copies the specified app to a specified project. The app should be one in a project that you can access, either an app which has been uploaded by a project member or a publicly availble app which has been copied to the project.

This call requires the following two arguments:

• project: include the project id
• name (optional): use this field to optionally re-name your app

aid <- a$public_app()[[1]]$id
a$copy_app(aid, project = pid, name = "copy-rename-test")
# check if it is copied
a$app(project = pid)

You can also copy directly from the app object, as shown below.

app <- a$public_app(id = "admin/sbg-public-data/rna-seq-alignment-star")
app$copy_to(project = "tengfei/api-testing",
            name = "copy of star")

3.10.3 Get an app’s CWL description

This call returns the raw CWL of a specific app, including its raw CWL. The call differs from the call to GET details of an app in that it returns a JSON object that is the CWL.

The app should be one in a project that you can access, either an app which has been uploaded by a project member or a publicly availble app which has been copied to the project.

To get a specific revision, pass the revision argument.

ap <- a$app(visibility = "public")[[1]]
a$project("demo")$app("index")
# get a specific revision
a$project("demo")$app("index", revision = 0)

Coming soon: converting apps to CWL objects

3.10.4 Add CWL as an app

To add a CWL object as an app, use the app_add function call for a Project object. The following two parameters are required:

• short_name: Project short names are based on the name you give to a project when you create it. Learn more about project short names on the Seven Bridges Platform or the CGC.
• filename: The name of the JSON file containing the CWL.

cwl.fl <- system.file("extdata", "bam_index.json", package = "sevenbridges")
a$project("demo")$app_add(short_name = "new_bam_index_app", filename = cwl.fl)
a$project("demo")$app_add(short_name = "new_bam_index_app", revision = 2, filename = cwl.fl)

Note: If you provide the same short_name, this will add a new revision.

3.10.5 Describe CWL in R directly

This is introduced in another vignette (vignette("apps", "sevenbridges")).

3.10.6 Create test or keep previous test for Tool

On sevenbridges platform, when you create or update your tools in the GUI, there is a test tab allow users to tweak the parameters and see what it looks like in your commandline simulatd terminal. To do this via R when you push your Tool object to your project, you need to provide “sbg:job” information like the example shown below

rbx <- Tool(id = "runif",
            label = "Random number generator",
            hints = requirements(docker(pull = "tengfei/runif"),
                                 cpu(1), mem(2000)),
            baseCommand = "runif.R",
            inputs = in.lst,
            outputs = out.lst,
            'sbg:job' = list(allocatedResources = list(mem = 9000, cpu = 1),
                             inputs = list(min = 1, max = 150)))
p$app_add("random", rbx)

Or if you have created a test info on the platform or previously pushed one, and you want to keep the previous test setup. We provide a arguments called keep_test to allow you keep the previous revision’s test information.

rbx <- Tool(id = "runif",
            label = "Random number generator",
            hints = requirements(docker(pull = "tengfei/runif"),
                                 cpu(1), mem(2000)),
            baseCommand = "runif.R",
            inputs = in.lst,
            outputs = out.lst)
p$app_add("random", rbx, keep_test = TRUE)

3.11.1 List tasks

This call returns a list of tasks that you can access. You are able to filter tasks by status.

# all tasks
a$task()
# filter
a$task(status = "completed")
a$task(status = "running")

To list all tasks in a project, use the following.

# a better way
a$project("demo")$task()

# alternatively
pid <- a$project("demo")$id
a$task(project = pid)

To list all tasks with details, pass detail = TRUE.

p$task(id = "your task id here", detail = TRUE)
p$task(detail = TRUE)

To list a batch task using the parent parameter, pass the batch parent task id.

p <- a$project(id = "tengfei/demo")
p$task(id = "2e1ebed1-c53e-4373-870d-4732acacbbbb")
p$task(parent = "2e1ebed1-c53e-4373-870d-4732acacbbbb")
p$task(parent = "2e1ebed1-c53e-4373-870d-4732acacbbbb", status = "completed")
p$task(parent = "2e1ebed1-c53e-4373-870d-4732acacbbbb", status = "draft")

3.11.2 Create a draft task

To create a draft task, you need to call the task_add function from the Project object. And you need to pass the following arguments:

• name: The name for this task
• description (optional): A description for this task
• app: The app id
• inputs: A list of inputs for this task

# push an app first
fl.runif <- system.file("extdata", "runif.json", package = "sevenbridges")
a$project("demo")$app_add("runif_draft", fl.runif)
runif_id <- "tengfei/demo-project/runif_draft"
# create a draft task
a$project("demo")$task_add(name = "Draft runif 3",
                           description = "Description for runif 3",
                           app = runif_id,
                           inputs = list(min = 1, max = 10))
# confirm
a$project("demo")$task(status = "draft")

3.11.3 Modify a task

Call the update function from a Task object to update the following:

• name
• description
• inputs list. Note that you can only update the items you provided.

# get the single task you want to update
tsk <- a$project("demo")$task("Draft runif 3")
tsk
tsk$update(name = "Draft runif update",
           description = "draft 2",
           inputs = list(max = 100))
# alternative way to check all inputs
tsk$getInputs()

3.11.4 Run a task

This call runs (“executes”) the specified task. Only tasks with a “DRAFT” status may be run.

tsk$run()
# run update without information just return latest information
tsk$update()

### Monitor a running task and set the hook function

To monitor a running task, call monitor from a Task object.

• First argument, set the interval time to check the status
• Rest arguments might be used for hook function

tsk$monitor()

Get and set default a hook function for task status. Currently, failed tasks will break the monitoring.

Note: Hook function has to return TRUE (break monitoring) or FALSE (continuing) in the end.

getTaskHook("completed")
getTaskHook("draft")
setTaskHook("draft", function(){message("never happens"); return(TRUE)})
getTaskHook("draft")

3.11.5 Abort a running task

This call aborts the specified task. Only tasks whose status is “RUNNING” may be aborted.

# abort
tsk$abort()
# check
tsk$update()

3.11.6 Delete a task

Note that you can only delete DRAFT tasks, not running tasks.

tsklst <- a$task(status = "draft")
# delete a single task
tsklst[[1]]$delete()
# confirm
a$task(status = "draft")
# delete a list of tasks
delete(tsklst)

3.11.7 Download all files from a completed task

tsk$download("~/Downloads")

3.11.8 Run task in batch mode

To run task in batch mode, (check ?batch) for more details. The code below is a mock.

# batch by items
(tsk <- p$task_add(name = "RNA DE report new batch 2",
                   description = "RNA DE analysis report",
                   app = rna.app$id,
                   batch = batch(input = "bamfiles"),
                   inputs = list(bamfiles = bamfiles.in,
                                 design = design.in,
                                 gtffile = gtf.in)))

# batch by metadata, input files has to have metadata fields specified
(tsk <- p$task_add(name = "RNA DE report new batch 3",
                   description = "RNA DE analysis report",
                   app = rna.app$id,
                   batch = batch(input = "fastq",
                                 c("metadata.sample_id", "metadata.library_id")),
                   inputs = list(bamfiles = bamfiles.in,
                                 design = design.in,
                                 gtffile = gtf.in)))

3.12.1 Create a volume

a <- Auth(user = "tengfei", platform = "aws-us")
a$add_volume(name   = "tutorial_volume",
             type   = "s3",
             bucket = "tengfei-demo",
             prefix = "",
             access_key_id     = "AKIAJQENSIA4DJQNZO3A",
             secret_access_key = "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",
             sse_algorithm     = "AES256",
             access_mode       = "RW")

3.12.2 List and search all volumes

# list all volume
a$volume()
# get unique volume by id
a$volume(id = "tengfei/tengfei_demo")
# partial search by name
a$volume(name = "demo")

3.12.3 Get a volume’s detail

v <- a$volume()
v[[1]]$detail()

3.12.4 Delete volume

This call deletes a volume you’ve created to refer to storage on Amazon Web Services or Google Cloud Storage.

Note that any files you’ve imported from your volume onto a Seven Bridges environment, known as an alias, will no longer be usable. If a new volume is created with the same volume_id as the deleted volume, aliases will point to files on the newly created volume instead (if those exist).

a$volume(id = "tengfei/tengfei_demo")$delete()

3.12.5 Import file from volume to project

This call imports a file from volume to your project.

v <- a$volume(id = "tengfei/tutorial_volume")
res <- v$import(location =  "A-RNA-File.bam.bai",
                project = "tengfei/s3tutorial",
                name = "new.bam.bai",
                overwrite = TRUE)

# get job status update
# state will be "COMPLETED" when it's finished, otherwise "PENDING"
v$get_import_job(res$id)
v

3.12.6 Export file from project to volume

Important :

• The file selected for export must not be a public file or an alias. Aliases are objects stored in your cloud storage bucket which have been made available on a Seven Bridges environment. The volume you are exporting to must be configured for read-write access. To do this, set the access_mode parameter to RW when creating or modifying a volume.
• If this call is successful, the original project file will become an alias to the newly exported object on the volume. The source file will be deleted from the Seven Bridges environment and, if no more copies of this file exist, it will no longer count towards your total storage price on the Seven Bridges environment.

When testing please update your file in a project.

res <- v$export(file = "579fb1c9e4b08370afe7903a",
                volume = "tengfei/tutorial_volume",
                location = "",  # when "" use old name
                sse_algorithm = "AES256")
# get job status update
# state will be "COMPLETED" when it's finished other wise "PENDING"
v$get_export_job(res$id)
v

3.13.1 Public files

# list the first 100 files
a$public_file()
# list by offset and limit
a$public_file(offset = 100, limit = 100)
# simply list everything!
a$public_file(complete = TRUE)
# get exact file by id
a$public_file(id = "5772b6f0507c175267448700")
# get exact file by name with exact = TRUE
a$public_file(name = "G20479.HCC1143.2.converted.pe_1_1Mreads.fastq", exact = TRUE)
# with exact = FALSE by default search by name pattern
a$public_file(name = "fastq")
a$public_file(name = "G20479.HCC1143.2.converted.pe_1_1Mreads.fastq")

Public files are hosted in the project called admin/sbg-public-data, and you can alternatively use the file request to get files you need.

3.13.2 Public apps

For public apps, there are similar API calls.

# list for 100 apps
a$public_app()
# list by offset and limit
a$public_app(offset = 100, limit = 50)
# search by id
a$public_app(id = "admin/sbg-public-data/control-freec-8-1/12")
# search by name in ALL apps
a$public_app(name = "STAR", complete = TRUE)
# search by name with exact match
a$public_app(name = "Control-FREEC", exact = TRUE, complete = TRUE)