---
title: "Introduction_2_query_variants"
author: "Hangjia Zhao"
date: "`r Sys.Date()`"
output: 
  BiocStyle::html_document:
  toc: true
vignette: >
  %\VignetteIndexEntry{Introduction_2_query_variants}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
    collapse = TRUE,
    comment = "#>"
)
```

[Progenetix](https://progenetix.org/) is an open data resource that provides curated individual cancer copy number variation (CNV) profiles along with associated metadata sourced from published oncogenomic studies and various data repositories. This vignette provides a comprehensive guide on accessing genomic variant data within the Progenetix database. 

If your focus lies in cancer cell lines, you can access data from [*cancercelllines.org*](https://cancercelllines.org/) by setting the `domain` parameter to `"cancercelllines.org"` in `pgxLoader` function. This data repository originates from CNV profiling data of cell lines initially collected as part of Progenetix and currently includes additional types of genomic mutations.

# Load library 

```{r setup}
library(pgxRpi)
library(SummarizedExperiment) # for pgxmatrix data
```

## `pgxLoader` function 

This function loads various data from `Progenetix` database via the Beacon v2 API with some extensions (BeaconPlus).

The parameters of this function used in this tutorial:

* `type`: A string specifying output data type. `"g_variants"` and `"cnv_fraction"` are used in this tutorial. 
* `output`: A string specifying output file format. The available options depend on the `type` parameter. When `type` is `"g_variants"`, available options are `NULL` (default), `"pgxseg"`, or `"seg"`; when `type` is `"cnv_fraction"`, available options are `NULL` (default) or `"pgxmatrix"`.
* `biosample_id`: Identifiers used in the query database for identifying biosamples. 
* `individual_id`: Identifiers used in the query database for identifying individuals. 
* `filters`: Identifiers used in public repositories, bio-ontology terms, or custom terms such as `"NCIT:C7376"`. For more information about filters, see the [documentation](https://docs.progenetix.org/common/beacon-api/#filters-filtering-terms).
* `codematches`: A logical value determining whether to exclude samples from child concepts 
of specified filters in the ontology tree. If `TRUE`, only samples exactly matching the specified filters will be included. Do not use this parameter when `filters` include ontology-irrelevant filters such as pubmed or cohort identifiers. Default is `FALSE`.
* `limit`: Integer to specify the number of returned profiles. Default is `0` (return all). 
* `skip`: Integer to specify the number of skipped profiles. E.g. if `skip = 2,limit=500`, the first 2*500=1000 profiles are skipped and the next 500 profiles are returned. Default is `0` (no skip).
* `dataset`: A string specifying the dataset to query from the Beacon response. Default is `NULL`, which includes results from all datasets.
* `save_file`: A logical value determining whether to save variant data as a local file instead of direct return. Only used when the parameter `type` is `"g_variants"`. Default is `FALSE`.
* `filename`: A string specifying the path and name of the file to be saved. Only used if the parameter `save_file` is `TRUE`. Default is `"variants.tsv"`, saved in the current working directory..
* `num_cores`: An integer specifying the number of cores to use for parallel processing during Beacon v2 variant data queries from multiple biosamples. Default is `1`.
* `use_https`: A logical value indicating whether to use the HTTPS protocol. If `TRUE`, the domain will be prefixed with `"https://"`; otherwise, `"http://"` will be used. Default is `TRUE`.
* `domain`: A string specifying the domain of the query data resource. Default is `"progenetix.org"`.
* `entry_point`: A string specifying the entry point of the Beacon v2 API. Default is `"beacon"`, resulting in the endpoint being `"https://progenetix.org/beacon"`.

# Retrieve variants 

Due to time-out constraints, variant data in Progenetix or cancercelllines can only be queried using biosample ids, rather than filters or individual ids. To speed up the process, you can use the `num_cores` parameter to enable parallel processing across multiple samples. For more details about retrieving biosample ids, refer to the vignette *Introduction_1_load_metadata*.

## Get biosample id 

```{r}
# get 2 samples for demonstration
biosamples <- pgxLoader(type="biosamples", filters = "NCIT:C3512", limit=2)

biosample_id <- biosamples$biosample_id

biosample_id
```

There are three output formats. 

## The first output format (by default)

The default output format extracts variant data from the Beacon v2 response, containing variant id and associated analysis id, biosample id and individual id. The CNV data is represented as [copy number change class](https://vrs.ga4gh.org/en/stable/concepts/SystemicVariation/CopyNumberChange.html#copy-change-term-definitions) following the GA4GH Variation Representation Specification (VRS).

```{r}
variant_1 <- pgxLoader(type="g_variants", biosample_id = biosample_id)
head(variant_1)
```

You can also query variant data from other Beacon v2 resources by setting the `domain` and `entry_point` parameters accordingly.

```{r}
variant_2 <- pgxLoader(type="g_variants", biosample_id = "cellzbs-kftvksak", domain = "cancercelllines.org", entry_point="beacon")
head(variant_2)
```

## The second output format (`output` = "pgxseg")

This format accesses data from [Progenetix services API](https://docs.progenetix.org/services/), which utilizes specialized resources within Progenetix. The '.pgxseg' file format contains segment mean values (in `log2` column), which are equal to log2(copy number of measured sample/copy number of control sample (usually 2)). A few variants are point mutations represented by columns `reference_bases` and `alternate_bases`.

```{r}
variant_3 <- pgxLoader(type="g_variants", biosample_id = biosample_id,output = "pgxseg")
head(variant_3)
```

## The third output format (`output` = "seg")

This format accesses data from [Progenetix services API](https://docs.progenetix.org/services/), which utilizes specialized resources within Progenetix. This format is compatible with IGV tool for visualization.

```{r}
variant_4 <- pgxLoader(type="g_variants", biosample_id = biosample_id,output = "seg")
head(variant_4)
```

# Export variants data for visualization

Setting `save_file` to TRUE in the `pgxLoader` function will save the retrieved variants data to a file 
rather than returning it directly. By default, the data will be saved in the current working directory, 
but you can specify a different path using the `filename` parameter. This export functionality is 
only available for variants data (when `type='g_variants'`).

## Upload 'pgxseg' file to Progenetix website

The following codes creates a '.pgxseg' file with the name "variants.pgxseg" in "~/Downloads/" folder. 

```{r eval=FALSE}
pgxLoader(type="g_variants", output="pgxseg", biosample_id=biosample_id, save_file=TRUE, 
          filename="~/Downloads/variants.pgxseg")
```

To visualize the '.pgxseg' file, you can either upload it to [this link](https://progenetix.org/service-collection/uploader) 
or use the [byconaut](https://byconaut.progenetix.org/) package for local visualization when dealing with a large number of samples.

## Upload '.seg' file to IGV

The following codes creates a special '.seg' file with the name "variants.seg" in "~/Downloads/" folder. 

```{r eval=FALSE}
pgxLoader(type="g_variants", output="seg", biosample_id=biosample_id, save_file=TRUE, 
          filename="~/Downloads/variants.seg")
```

You can upload this '.seg' file to [IGV tool](https://software.broadinstitute.org/software/igv/) for visualization.

# Retrive CNV fraction of biosamples 

Because segment variants are not harmonized across samples, Progenetix provides processed CNV features, known as CNV fractions. These fractions represent the proportion of genomic regions overlapping one or more CNVs of a given type, facilitating sample-wise comparisons. The following query is based on filters, but biosample id and individual id are also available for sample-specific CNV fraction queries. For more information about filters, biosample id and individual id, as well as the use of parameters `skip`, `limit`, and `codematches`, see the vignette *Introduction_1_load_metadata*.

CNV fraction data is retrieved through the [Progenetix services API](https://docs.progenetix.org/services/). When making a query, the domain of the data resource should be set to either `"progenetix.org"` (by default) or `"cancercelllines.org"`.

## Across chromosomes or the whole genome 

```{r}
cnv_fraction <- pgxLoader(type="cnv_fraction", filters = "NCIT:C2948")
```

This includes CNV fraction across chromosome arms, whole chromosomes, or the whole genome.

```{r}
names(cnv_fraction)
```

The CNV fraction across chromosomal arms looks like this

```{r}
head(cnv_fraction$arm_cnv_frac)[,c(1:4, 49:52)]
```

The row names are analyses ids from samples that belong to the input filter NCIT:C2948. There are 96 columns. 
The first 48 columns are duplication fraction across chromosomal arms, followed by deletion fraction. 
CNV fraction across whole chromosomes is similar, with the only difference in columns.

The CNV fraction across the genome (hg38) looks like this

```{r}
head(cnv_fraction$genome_cnv_frac)
```

The first column is the total called fraction, followed by duplication fraction and deletion fraction.

## Across genomic bins 

```{r}
cnvfrac_matrix <- pgxLoader(type="cnv_fraction", output="pgxmatrix", filters = "NCIT:C2948")
```

The returned data is stored in [*RangedSummarizedExperiment*](https://bioconductor.org/packages/release/bioc/html/SummarizedExperiment.html) object, which is a matrix-like container where rows represent ranges of interest (as a GRanges object) and columns represent analyses derived from biosamples. The data looks like this

```{r}
cnvfrac_matrix
```

You could get the interval information by `rowRanges` function from *SummarizedExperiment* package.

```{r}
rowRanges(cnvfrac_matrix)
```

To access the CNV fraction matrix, use `assay` accesssor from *SummarizedExperiment* package

```{r}
assay(cnvfrac_matrix)[1:3,1:3]
```

The matrix contains 6212 rows, corresponding to genomic regions. These rows include 3,106 intervals with "gain status" and 3,106 intervals with "loss status". The columns represent analysis profiles derived from samples that belong to the input filter.

The value is the fraction of the binned interval overlapping with one or more CNVs of the given type (DUP/DEL). For example, if the value in the second row, the first column is 0.2, it means that one or more duplication events overlapped with 20% of the genomic bin located in chromosome 1: 400000-1400000 in the first analysis profile.

To get associated biosample id and filters for analyses, use `colData` function from *SummarizedExperiment* package:

```{r}
colData(cnvfrac_matrix)
```

It is noted that the number of analysis profiles does not necessarily equal the number of samples. One biosample id may correspond to multiple analysis ids. `group_id` corresponds to the meaning of `filters`.

# Session Info

```{r echo = FALSE}
sessionInfo()
```
