Understand your app input

Now assume you already copied the public app “admin/sbg-public-data/rna-seq-alignment-star/5” into your project “tengfei/api-testing” , the app id in your current project is “tengfei/api-testing/newcopyofstar” Or if you already have an app to run in your project.

To draft a new task, you need to specify

• name of the task
• description
• app id
• inputs of your task: you need to know about the app you are running, in this case, the cwl app accept 4 parameters (number, min, max, seed).

You can always go to the UI to check App details or task input requirements, but how to do it in R?

To check your inputs name and types you need to get an App object first.

app = a$app(id = "tengfei/api-testing-2/newcopyofstar")
## get input matrix
app$input_matrix()
app$input_matrix(c("id", "label", "type"))
app$input_matrix(c("id", "label", "type"), required = TRUE)
## get required input names and types only
app$get_required()

Or loaded from a CWL JSON and convert it into a R object first.

f1 = system.file("extdata/app", "flow_star.json", package = "sevenbridges")
app = convert_app(f1)
## get input matrix
app$input_matrix()

##                                id                    label    type
## 1                    #sjdbGTFfile              sjdbGTFfile File...
## 2                          #fastq                    fastq File...
## 3               #genomeFastaFiles         genomeFastaFiles    File
## 4 #sjdbGTFtagExonParentTranscript      Exons' parents name  string
## 5       #sjdbGTFtagExonParentGene                Gene name  string
## 6          #winAnchorMultimapNmax         Max loci anchors     int
## 7             #winAnchorDistNbins Max bins between anchors     int
##   required fileTypes
## 1    FALSE      null
## 2     TRUE      null
## 3     TRUE      null
## 4    FALSE      null
## 5    FALSE      null
## 6    FALSE      null
## 7    FALSE      null

app$input_matrix(c("id", "label", "type"))

##                                id                    label    type
## 1                    #sjdbGTFfile              sjdbGTFfile File...
## 2                          #fastq                    fastq File...
## 3               #genomeFastaFiles         genomeFastaFiles    File
## 4 #sjdbGTFtagExonParentTranscript      Exons' parents name  string
## 5       #sjdbGTFtagExonParentGene                Gene name  string
## 6          #winAnchorMultimapNmax         Max loci anchors     int
## 7             #winAnchorDistNbins Max bins between anchors     int

app$input_matrix(c("id", "label", "type"), required = TRUE)

##                  id            label    type
## 2            #fastq            fastq File...
## 3 #genomeFastaFiles genomeFastaFiles    File

## get required input names and types only
app$get_required()

##            fastq genomeFastaFiles 
##        "File..."           "File"

For what it returned, the names is the one you want to use as your task input list. Task inputs need to match the expected data type and name, requried has to be provided. In above example, we see two required field

• fastq: File array as indicated by “File…”
• genomeFastaFiles: single file as indicated by “File”

We also want to provide gene feature file

• sjdbGTFfile: single file as indicated by “File”

Note some other input types includes:

• number, character or integer, you can directly pass to input parameters as it is.
• an Enum type, just pass the value to the input parameter
• File, you have to pay attention to this. Some parameters accept only single file type ‘File’ as input while other input takes more than one files (File arrays or FilesList or ‘File…’ ). This input require you to pass a Files object (for sinlge file input) or FilesList object (for input accept more than one files) or simply a list of “Files” object. Get your file by id or by name with exact = TRUE is always accurate. Sounds complicated let’s just see an example.

Get your input files ready

fastqs <- c("SRR1039508_1.fastq", "SRR1039508_2.fastq")

## get all 2 exact files
fastq_in = p$file(name= fastqs, exact = TRUE) 

## get single file
fasta_in = p$file(name = "Homo_sapiens.GRCh38.dna.primary_assembly.fa", 
                  exact = TRUE)
## get all single file
gtf_in = p$file(name = "Homo_sapiens.GRCh38.84.gtf", 
                exact = TRUE)

Draft a new task

## Add new tasks
taskName = paste0("tengfei_star-alignment ",date())

tsk = p$task_add(name = taskName,
           description = "star test", 
           app = "tengfei/api-testing-2/newcopyofstar/0",
           inputs = list(sjdbGTFfile = gtf_in,
                         fastq = fastq_in,
                         genomeFastaFiles = fasta_in))

Remember “fastq” expect a list of files. You can also do something like

f1 = p$file(name = "SRR1039508_1.fastq", exact = TRUE)
f2 = p$file(name = "SRR1039508_2.fastq", exact = TRUE)
## get all 2 exact files
fastq_in = list(f1, f2) 

## or if you know you only have 2 file name matching SRR924146*.fastq
fastq_in = p$file(name = "SRR1039508*.fastq", complete = TRUE)

Using complete = TRUE when items is over 100.

Draft a batch task

Now let’s do a batch with 8 files in 4 group, which is batch by metadata sample_id and library_id, assume each file has these two metadata fields in the system. And it could be evenly grouped into 4. So we will have a single parent task with 4 sub tasks.

fastqs <- c("SRR1039508_1.fastq", "SRR1039508_2.fastq", "SRR1039509_1.fastq", 
            "SRR1039509_2.fastq", "SRR1039512_1.fastq", "SRR1039512_2.fastq", 
            "SRR1039513_1.fastq", "SRR1039513_2.fastq")

## get all 8 files
fastq_in = p$file(name= fastqs, exact = TRUE) 
## can also try to returned all SRR*.fastq files
## fastq_in = p$file(name= "SRR*.fastq", complete = TRUE) 

tsk = p$task_add(name = taskName,
           description = "Batch Star Test", 
           app = "tengfei/api-testing-2/newcopyofstar/0",
           batch = batch(input = "fastq", 
                         criteria = c("metadata.sample_id","metadata.noexist_id")),
           inputs = list(sjdbGTFfile = gtf_in,
                         fastq = fastqs_in,
                         genomeFastaFiles = fasta_in))

Now you have a draft batch task, please check it out in the UI.

Set up default token for different platforms

You can create a file called ‘.sbg.auth.yml’ in your home folder, and maintain multiple account for a list of platforms, including private or public ones.

us:
  url: https://api.sbgenomics.com/v2/
  user:
    tengfei:
      token: fake_token
    yintengfei:
      token: fake_token
cgc:
  url: https://cgc-api.sbgenomics.com/
  user:
    tengfei:
      token: fake_token
gcp:
  url: https://gcp-api.sbgenomics.com/v2/
  user:
    tengfei:
      token: fake_token

When you load sevenbridges package, it will first try to parse your token configuration file first into an options list.

## Create Auth object from config file
a <- Auth(username = "yintengfei", platform = "us")
## show all 
getToken()
## show all pre-set user token for platform 
getToken("cgc")
## show individual token for a user
getToken(platform = "cgc", username = "tengfei")

Note: when you edit your .sbg.auth.yml, you have to reload your package.

Create Auth object directly

First thing first, you need to construct an Auth object, everything begins with this object, it stores

• The authentication token
• The API URL
• The platform (US platform, Cancer Genomics Cloud etc), this is optional, will translate into API url.

The logic is like this

1. If you didn’t pass url or token, we think you are loading from config file
2. if no platform or user provided, will use the first item in your token config file, this is not recommended, at least provide platform/username set.

library(sevenbridges)
## direct setup
a <- Auth(token = "1c0e6e202b544030870ccc147092c257",
          url = "https://cgc-api.sbgenomics.com/v2/")

By default it points to Cancer Genomics Cloud platform, unless you specify

• API URL (more flexible)
• or Platform (currently support ‘cgc’, ‘us’, ‘gcp’)

Note: when you construct the Auth object, make sure you input the correct platform or API url for your authentication. On Seven Bridges related platforms, you can always find it under your account setting and developer tab.

For the tutorial about how to get your authentication, please check

• Get token from US platform
• Get token from Cancer Genomics Cloud Platform

For billing

if no id provided, This call returns a list of paths used to access billing information via the API. else, This call lists all your billing groups, including groups that are pending or have been disabled. if breakdown = TRUE, This call returns a breakdown of spending per-project for the billing group specified by billing_group. For each project that the billing group is associated with, information is shown on the tasks run, including their initiating user (the runner), start and end times, and cost.

## return a BillingList object
(b <- a$billing())
a$billing(id = b$id, breakdown = TRUE)

For invoices

If no id provided, This call returns a list of invoices, with information about each, including whether or not the invoice is pending and the billing period it covers. The call returns information about all your available invoices, unless you use the query parameter bg_id to specify the ID of a particular billing group, in which case it will return the invoice incurred by that billing group only. if id provided, This call retrieves information about a selected invoice, including the costs for analysis and storage, and the invoice period.

a$invoice()
a$invoice(id = "fake_id")

Note (TODO): Invoice is not an object yet, it currently just return a list.

List All Projects

This call returns a list of all projects you are a member of. Each project’s project_id and URL on the CGC will be returned.

a$project()

Then if you want to list the projects owned by and accessible to a particular user, specify the owner argument. Each project’s ID and URL will be returned.

a$project(owner = "tengfei")
a$project(owner = "yintengfei")

To get details about project(s), use detail = TRUE

a$project(detail = TRUE)

Partial Match Project Name

For more friendly interface and convenient search, we support partial name match in this interface. The first argument for the call is “name”, users can provide part of the name and we do a search for you automatically.

## want to return a project called
a$project("hello")

Create a New Project

To create a new project, user need to specify

• name (required)
• billing_group_id (required)
• description (optional)
• tags (optional): this has to be a list(), only if you are “TCGA” user, you can create TCGA project by passing tags list(“tcga”)
• type (optional): by default, we are creating a cwl project “v2”

a$project_new("api_testing_tcga", b$id,
              description = "Test for API")

Create a new project with TCGA controlled data on CGC

Just need to pass a “tags” list with value “tcga”

a$project_new("controlled_project", b$id,
              description = "Test for API", tags = list("tcga"))

Delete a Project

Next we delete what we created for testing, only single project could be deleted now by call $delete(), so please pay attention to the returned object from a$project(), sometimes if you are using partial matching by name, it will return a list. If you want to operate on a list of object, we provide some batch function, please read relevant section.

## remove it, not run
a$project("api_testing")$delete()
## check
## will delete all projects matcht the name
delete(a$project("api_testing_donnot_delete_me"))

Update/Edit a Project

You can update information about an existing project, including

• name
• description
• billing_group

a$project(id = "tengfei/helloworld")
a$project(id = "tengfei/helloworld")$update(name = "Hello World Update", 
                                                    description = "Update description")

Project Member

a$project(id = "tengfei/demo-project")$member()

m <- a$project(id = "tengfei/demo-project")$member_add(username = "yintengfei")

Update a member

This call edits a user’s permissions in a specified project. It can only be successfully made by a user who has admin permissions in the project.

m <- a$project(id = "tengfei/demo-project")$
       member(username = "yintengfei")
m$update(copy = TRUE)

== Member ==
username : yintengfei
-- Permission --
read : TRUE
write : FALSE
copy_permission : TRUE
execute : FALSE
admin : FALSE

m$delete()
## confirm
a$project(id = "tengfei/demo-project")$member()

List all Files

To list all files belongs to a project simple use

p <- a$project(id = "tengfei/demo-project")
p$file()

List all files

A better way to list files in a project

## first 100 files, default offset = 0, limit = 100
p$file()
## list all files
p$file(complete = TRUE)

Alternatively get files from Auth object.

a$file(project = p$id)
a$file(name = "omni", project = p$id, detail = TRUE)

Search and filter file(s)

p = a$project(id = "tengfei/demo")
p$file(metadata = list(sample_id = "Sample1", 
                       sample_id = "Sample2", 
                       library_id = "EXAMPLE"), 
       tag = c("hello", "world"))

## return single object id is "some_file_id"
p$file(id = "some_file_id")
## return single object named a.fastq
p$file(name = "a.fastq", exact = TRUE)
## public file search using Auth object
a$public_file(name = "ucsc.hg19.fasta.fai", exact = TRUE)
a$public_file(id = "578cf94a507c17681a3117e8")

## get two files
p$file(name = c("test1.fastq", "test2.fastq"), exact = TRUE)
## get two files from public files using shorhand
a$public_file(name = c("ucsc.hg19.fasta.fai", "ucsc.hg19.fasta"), 
              exact = TRUE)

## get matchd the pattern for searching first 100 files
p$file(name = c("gz", "fastq"))
## get all matched files from the project
p$file(name = c("gz", "fastq"), complete = TRUE)
## get all files matched ucsc
a$public_file(name = "ucsc.hg19", complete = TRUE)

p = a$project(id = "tengfei/demo")
p$file(metadata = list(sample_id = "Sample1", 
                       sample_id = "Sample2", 
                       library_id = "EXAMPLE")) 

p = a$project(id = "tengfei/demo")
p$file(tag = c("s1", "s2")) 

## list all outputs file from a task id
a$task(id = "53020538-6936-422f-80de-02fa65ae4b39")$file()

## OR
p = a$project(id = "tengfei/demo")
p$file(origin.task = "53020538-6936-422f-80de-02fa65ae4b39")

Copy a file or group of files

This call copies the specified file to a new project. Files retain their metadata when copied, but may be assigned new names in their target project.

Note that Controlled Data files may not be copied to Open Data projects. To make this call, you should have copy permission within the project you are copying from.

Let’s try to copy a file from CGC public files, the id you can tell from the url is “561e1b33e4b0aa6ec48167d7”

You must provide

• id file id, or list/vector of files ids.
• project parameter: project id.
• name is optional, if omitted, use the same.

## 1000G_omni2.5.b37.vcf
fid <- "561e1b33e4b0aa6ec48167d7"
fid2 <- "561e1b33e4b0aa6ec48167d3"
pid <- a$project("demo")$id
a$copyFile(c(fid, fid2), project = pid)
a$project(id = pid)$file()

NOTE: to copy a group of files, you need Auth$copyFile() interface. The id of those files in your project will be different from public id.

Alternatively you can do single file copy like this

a$project("hello")$file(id = fid)$copyTo(pid)

Delete file(s)

Note: the delete action only work for single file now, make sure your file call return a single file not a file list.

a$project("demo")$file()[[1]]$delete()
## confirm the deletion
a$project("demo")$file()

You can also delete a group of files or FilesList object, be careful with this function!

## return 5 files
a$project("demo")$file("phase1")
## delete all of them
delete(a$project("demo")$file("phase1"))
a$project("demo")$file("phase1")

Download files

To get the download information, basically a url, please use

a$project("demo")$file()[[1]]$download_url()

To download directly from R, use download call directly from single File object.

fid <- a$project("demo")$file()[[1]]$id
a$project("demo")$file(id = fid3)$download("~/Downloads/")

I also created download function for FilesList object to save your time

fls <- a$project("demo")$file()
download(fls, "~/Downloads/")

To download all files from a project.

a$project("demo")$download("~/Downloads")

Upload files

Seven Bridges platforms provide couple different ways for data import

• command line uploader
• graphic UI uploader
• from ftp, http etc from interface directly
• api uploader that you can directly call with sevenbridges package

API client uploader is working like this, simply call project$upload function to upload a file a file list or a folder recursively…

a <- Auth(username = "tengfei", platform = "cgc")
fl <- system.file("extdata", "sample1.fastq", package = "sevenbridges")

(p <- a$project(id = "tengfei/quickstart"))

## by default load .meta for the file
p$upload(fl, overwrite = TRUE)
## pass metadata
p$upload(fl, overwrite = TRUE, metadata = list(library_id = "testid2", platform = "Illumina x11"))
## rename
p$upload(fl, overwrite = TRUE, name = "sample_new_name.fastq", 
         metadata = list(library_id = "new_id"))

Upload a folder

dir.ext <- system.file("extdata", package = "sevenbridges")
list.files(dir.ext)
p$upload(dir.ext, overwrite = TRUE)

Upload a file list

dir.ext <- system.file("extdata", package = "sevenbridges")
## enable full name
fls <- list.files(dir.ext, recursive = TRUE, full.names = TRUE)
p$upload(fls, overwrite = TRUE)
p$upload("~/Documents/Data/sbgtest/1000G_phase1.snps.high_confidence.b37.vcf")

Update a file

You can call update() function from Files object, following things could be updated

• name
• metadata (list): this is going to overwrite all meta for the file, so please provide the full list. For more flexible operation, please check next section about Metadata.

If no parameters provided, will just get detail for the same file and update the object itself.

(fl <- a$project(id = "tengfei/demo-project")$file(name = "sample.fastq"))

== File ==
id : 56c7916ae4b03b56a7d7
name : sample.fastq
project : tengfei/demo-project

Show metadata

## show metadata
fl$meta()

Update meta

fl$update(name = "sample.fastq", metadata = list(new_item1 = "item1", new_item2 = "item2", file_extension = "fastq"))
## check it out
fl$meta()

Metadata Operation

A full list of metadata fields and their permissible values on the CGC is available on the page TCGA Metadata.

Note that the file name is not the same as its ID. The ID is a hexadecimal string, automatically assigned to a file in a project. The file’s name is a human-readable string. For information, please see the API overview.

To get metadata for a file call meta().

## meta is pulling the latest information via API
fl$meta()
## field meta data saved the previous saved one
fl$metadata

Although CGC defined a set of meta schema, which is visible on the UI of the platform, but you can pass any free form of meta for the file, it’s just not visible on UI, but it’s stored with the data.

Only the value being specified stored with files, to set metadata please call set_meta() from Files object.

Important:

• By default, we are not overwriting the meta field using set_meta call, unless you pass the overwrite = TRUE argument

fl$set_meta(new_item3 = "item3")
fl
## oops it removed rest of the meta
fl$set_meta(new_item4 = "item4", overwrite = TRUE)
fl

Let’s keep playing with meta, if you are really interested in the default schema that shown on the UI, you can use Metadata() constructor and check details of each meta; Simply call the function (name of meta), it will show description and enumerated items. Please pay attention to suggested_values field.

## check which schema we have
Metadata()$show(full = TRUE)
## check details for each, play with it
platform()
paired_end()
quality_scale()

You can see some have suggested value, to construct the Metadata, we encourage you use Metadata() directly, pass metadata directly into the call, it will do the validation.

Metadata(platform  = "Affymetrix SNP Array 6.0", 
         paired_end = 1, 
         quality_scale = "sanger",
         new_item = "new test")

Tag for file(s)

Tags are different from metadata, it’s more convenient and flexible and visible from file list UI on the platform.

To add a new tag using obj$add_tag() method for File object with a single tag string or a list/vector of tags. To complete rewrite the tags for file, use obj$set_tag() method, by default the argument overwrite = TRUE which means swipe and reset, when it’s set to FALSE, it is basically add_tag call. You can also filter files by tags.

Let’s get a file called “sample.bam” and play with it’s tag.

p <- a$project(id = "tengfei/s3tutorial")
fl = p$file("sample.bam", exact = TRUE)
## show tags for single file
fl$tag()
## add new tags
fl$add_tag("new tag")
## equavilent to 
fl$set_tag("new tag 2", overwrite = FALSE)
## set tags to overwrite existing
x = list("this", "is", 1234)
fl$set_tag(x)
## filter by tags
p$file(tag = c("1234", "new"))
p$file(tag = list("1234", "new"))
p$file(tag = "1234")

We also provide convenient methods for FilesList, let’s add tag “s1” to a group files matched “Sample1” and add tag “s2” to a group of files that matched “s2”, then try to get files by tag filter.

## work on a group of files
## add tag "s2" to a group of files named with "Sample2" in it
fl2 = p$file("Sample2")
add_tag(fl2, "s2")
## add tag "s2" to a group of files named with "Sample1" in it
fl1 = p$file("Sample1")
add_tag(fl1, "s1")
## filter by tag s1 or s2
p$file(tag = "s1")
p$file(tag = "s2")
## get files tagged with s2 and 1234
p$file(tag = list("s2", "s1"))

How to do that on the UI, please check out the tutorial here

List all apps

This call lists all the apps available to you.

a$app()
## or show details
a$app(detail = TRUE)

To search a name, please pass a pattern for the name argument; or provide a unique id.

## pattern match
a$app(name = "STAR")
## unique id
aid <- a$app()[[1]]$id
aid
a$app(id = aid)
## get a specific revision from an app
a$app(id = aid, revision = 0)

To list all apps belong to one project use project argument

## my favorite, always
a$project("demo")$app()

## or alternatviely
pid <- a$project("demo")$id
a$app(project = pid)

To list all public apps, use visibility argument

## show 100 items from public
x = a$app(visibility = "public")
length(x)
x = a$app(visibility = "public", complete = TRUE)
length(x)
x = a$app(project = "tengfei/helloworld", complete = TRUE)
length(x)
a$app(visibility = "public", limit = 5, offset = 150)

To search an app cross all published apps (this may take a while)

a$app("STAR", visibility = "public", complete = TRUE)

Copy an App

This call copies the specified app to the specified project. The app should be one in a project that you can access; this could be an app that has been uploaded to the CGC by a project member, or a publicly available app that has been copied to the project.

Need two arguments

• project: id character
• name: optional, to re-name your app

aid <- a$public_app()[[1]]$id
a$copy_app(aid, project = pid, name = "copy-rename-test")
## check it is copied
a$app(project = pid)

Or you can copy directly from an app object

app = a$public_app(id = "admin/sbg-public-data/rna-seq-alignment-star")
app$copy_to(project = "tengfei/api-testing", 
            name = "copy of star")

Get CWL from an App

This call returns information about the specified app, as raw CWL. The call differs from the call to GET details of an app by returning a JSON object that is the CWL.

The app should be one in a project that you can access; this could be an app that has been uploaded to the CGC by a project member, or a publicly available app that has been copied to the project.

To get a specific revision, pass revision argument.

ap <- a$app(visibility = "public")[[1]]
a$project("demo")$app("index")
## get a specific revision
a$project("demo")$app("index", revision = 0)

TODO: convert it to an CWL object

Add CWL as an APP

Use app_add function call from a Project object, two parameters required

• short_name: a short id for your app, alphanumeric character, no spacing; this not name field.
• filename: you json file for cwl.

cwl.fl <- system.file("extdata", "bam_index.json", package = "sevenbridges")
a$project("demo")$app_add(short_name = "new_bam_index_app", filename = cwl.fl)
a$project("demo")$app_add(short_name = "new_bam_index_app", revision = 2, filename = cwl.fl)

Note: provide the same short_name will add new revision

Directly Describe CWL in R

This is fun and is introduced in another vignette.

List tasks

This call returns a list of tasks that you can access. You are able to filter by status

## all tasks
a$task()
## filter
a$task(status = "completed")
a$task(status = "running")

To list all tasks in a project

## better way
a$project("demo")$task()

## alternatively
pid <- a$project("demo")$id
a$task(project = pid)

To list all tasks with details just pass detail = TRUE.

p$task(id = "your task id here", detail = TRUE)
p$task(detail = TRUE)

To list a batch task using parent parameter, pass the batch parent task id.

p = a$project(id = "tengfei/demo")
p$task(id = "2e1ebed1-c53e-4373-870d-4732acacbbbb")
p$task(parent = "2e1ebed1-c53e-4373-870d-4732acacbbbb")
p$task(parent = "2e1ebed1-c53e-4373-870d-4732acacbbbb", status = "completed")
p$task(parent = "2e1ebed1-c53e-4373-870d-4732acacbbbb", status = "draft")

Create a draft task

To create a draft, you need to call the task_add function from Project object. And you need to pass following arguments

• name: name for this task
• description: description for this task
• app: app id you have access to
• inputs: inputs list for this task

## push an app first
fl.runif <- system.file("extdata", "runif.json", package = "sbgr")
a$project("demo")$app_add("runif_draft", fl.runif)
runif_id <- "tengfei/demo-project/runif_draft"
## create a draft task
a$project("demo")$task_add(name = "Draft runif 3", 
                           description = "Description for runif 3", 
                           app = runif_id,
                           inputs = list(min = 1, max = 10))
## confirm
a$project("demo")$task(status = "draft")

Modify a task

Call update function fro a Task object, you can update

• name
• description
• inputs list (only update items you provided.)

## get the single task you want to update
tsk <- a$project("demo")$task("Draft runif 3")
tsk
tsk$update(name = "Draft runif update", description = "draft 2", 
           inputs = list(max = 100))
## alternative way to check all inputs
tsk$getInputs()

Run a task

This call runs (executes) the specified task. Only tasks whose status is “DRAFT” may be run.

tsk$run()
## run update without information just return latest information
tsk$update()

Monitor a running task and set function hook

To monitor a running task, call monitor from a task object

• first argument set interval time to check the status
• rest arguments might be used for hook function

tsk$monitor()

get and set default hook function for task status, currently failed, completed tasks will break the monitoring.

Note: Hook function has to return TRUE (break monitoring) or FALSE (continuing) in the end.

getTaskHook("completed")
getTaskHook("draft")
setTaskHook("draft", function(){message("never happens"); return(TRUE)})
getTaskHook("draft")

Abort a runing task

This call aborts the specified task. Only tasks whose status is “RUNNING” may be aborted.

## abort
tsk$abort()
## check
tsk$update()

Delete a task

Note that you can only delete draft tasks, not running tasks.

tsklst <- a$task(status = "draft")
## delete a single task
tsklst[[1]]$delete()
## confirm
a$task(status = "draft")
## delete a list of tasks
delete(tsklst)

Download all files from a completed task

tsk$download("~/Downloads")

Run task in bacth mode

To run task in batch mode, (check ?batch) for more details, here is an mock running

## batch by items
(tsk <- p$task_add(name = "RNA DE report new batch 2", 
                   description = "RNA DE analysis report", 
                   app = rna.app$id,
                   batch = batch(input = "bamfiles"),
                   inputs = list(bamfiles = bamfiles.in, 
                                 design = design.in,
                                 gtffile = gtf.in)))

## batch by metadata, input files has to have metadata fields specified
(tsk <- p$task_add(name = "RNA DE report new batch 3", 
                   description = "RNA DE analysis report", 
                   app = rna.app$id,
                   batch = batch(input = "fastq", 
                                 c("metadata.sample_id", "metadata.library_id")),
                   inputs = list(bamfiles = bamfiles.in, 
                                 design = design.in,
                                 gtffile = gtf.in)))

Create a volume

a = Auth(user = "tengfei", platform = "us")
a$add_volume(name = "tutorial_volume", 
             type = "s3", 
             bucket = "tengfei-demo", 
             prefix = "",
             access_key_id = "AKIAJQENSIA4DJQNZO3A",
             secret_access_key = "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", 
             sse_algorithm = "AES256", 
             access_mode = "RW")

List and search all volume

## list all volume
a$volume()
## get unique volume by id
a$volume(id = "tengfei/tengfei_demo")
## partial search by name
a$volume(name = "demo")

Get volume detail

v = a$volume()
v[[1]]$detail()

Delete volume

a$volume(id = "tengfei/tengfei_demo")$delete()

Import file from volume to project

This call import a file from volume like s3 bucket to your project.

v = a$volume(id = "tengfei/tutorial_volume")
res = v$import(location =  "A-RNA-File.bam.bai",
         project = "tengfei/s3tutorial",
         name = "new.bam.bai", 
         overwrite = TRUE)

## get job status update
## state will be "COMPLETED" when it's finished other wise "PENDING"
v$get_import_job(res$id)
v

Export file from project to volume

Important :

• The file selected for export must not be a public file or an alias. Aliases are objects stored in your cloud storage bucket which have been made available on the Platform. The volume you are exporting to must be configured for read-write access. To do this, set the access_mode parameter to RW when creating or modifying a volume.
• If this call is successful, the original project file will become an alias to the newly exported object on the volume. The source file will be deleted from the Platform and, if no more copies of this file exist, it will no longer count towards your total storage price on the Platform.

When test please update your file to a project.

res = v$export(file = "579fb1c9e4b08370afe7903a", 
         volume = "tengfei/tutorial_volume",
         location = "", ## when "" use old name
         sse_algorithm =  "AES256")
## get job status update
## state will be "COMPLETED" when it's finished other wise "PENDING"
v$get_export_job(res$id)
v

Public files

## list first 100 files 
a$public_file()
## list by offset and limit
a$public_file(offset = 100, limit = 100)
## simply list everything!
a$public_file(complete = TRUE)
## get exact file by id
a$public_file(id = "5772b6f0507c175267448700")
## get exact file by name with exact = TRUE
a$public_file(name = "G20479.HCC1143.2.converted.pe_1_1Mreads.fastq", exact = TRUE)
## with exact = FALSE by default search by name pattern
a$public_file(name = "fastq") 
a$public_file(name = "G20479.HCC1143.2.converted.pe_1_1Mreads.fastq")

Actually the public files are hosted in the project called “admin/sbg-public-data”, so of course you can just use file api to get files you need.

Public apps

For public apps we have the similar API calls.

## list for 100 apps
a$public_app()
## list by offset and limit
a$public_app(offset = 100, limit = 50)
## search by id
a$public_app(id = "admin/sbg-public-data/control-freec-8-1/12")
## search by name in ALL apps
a$public_app(name = "STAR", complete = TRUE)
## search by name with exact match
a$public_app(name = "Control-FREEC", exact = TRUE, complete = TRUE)