Metadata are crucial to all forms of statistical analysis. Metadata are used to define formal steps of upstream data preprocessing, to annotate outcomes and covariates, and to interpret inferential results.
The ssrch package was developed to provide a lightweight approach to searching a metadata corpus with R. There are three basic steps:
We illustrate the system using a collection of tables of metadata about cancer transcriptomics.
To provide a sense of what is at stake, we work with 68 CSV files
derived from the NCBI Sequence Read Archive metadata.
The files consist of contents of the sample.attribute
subtable of study metadata
retrieved using the SRAdbV2 package at github.com:seandavi/SRAdbV2.
The CSV files
are lodged in a zip file
canc68.zip
with the ssrch
package in inst/cancer_corpus
.
The CSV files have been indexed in an S4 object of class
DocSet
.
## ssrch DocSet resource:
## 68 documents, 9934 records
## some titles:
## Single cell analysis of lung adenocarcinoma cell lines ... (DRP001358)
## Single cell analysis of lung adenocarcinoma cell lines ... (DRP002586)
## Integrative systematic analyses of mutational and trans... (ERP010142)
## Identification_and_molecular_charaterization_of_new_tum... (ERP012527)
## Whole transcriptome profiling of Esophageal adenocarcin... (ERP013206)
## Transcriptional landscape of human tissue lymphocytes u... (ERP013260)
A single document can be retrieved with the retrieve_doc
function, given the study accession number.
## X study.accession experiment.accession Bcl.2 Colletion.date
## 1 1 ERP010142 ERX943467 Negative 2001-10-10
## 2 2 ERP010142 ERX943327 Positive 2005-12-23
## 3 3 ERP010142 ERX943378 Negative 2001-07-11
There is partial standardization of field names in this corpus, but there is considerable variation among studies in the number and types of fields used.
docids = ls(envir=docs2kw(docset_cancer68))
allc = lapply(docids,
function(x) try(retrieve_doc(x, docset_cancer68)))
summary(sapply(allc,ncol))
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 4.000 6.000 8.000 9.221 10.000 25.000
## [[1]]
## [1] "X" "study.accession" "experiment.accession"
## [4] "source_name" "tissue" "tissue.archive.method"
## [7] "cell.line" "cell.type" "development.stage"
## [10] "disease.status"
##
## [[2]]
## [1] "X" "study.accession" "experiment.accession"
## [4] "source_name" "tissue" "cell.type"
## [7] "cell.activation" "cell.category" "barcode"
The ssrch
package provides
tools for identifying studies in which a given
field is used, and in which given values are recorded.
A motivating example is determining what studies involve measurements on normal tissue adjacent to tumor. In our collection, this can be assessed via:
## hits docs
## 1 Adjacent SRP056696
## 2 Adjacent SRP111343
## 3 Adjacent Normal tissue SRP056696
## 4 Adjacent normal SRP111343
## 5 adjacent SRP069212
## 6 adjacent SRP073447
## 7 adjacent normal SRP069212
## 8 adjacent normal SRP073447
## 9 benign_adjacent SRP011439
Before getting into the details of search engine construction, we review some high-level concepts and methods.
First, we have the container for managing our search resource.
## Class "DocSet" [package "ssrch"]
##
## Slots:
##
## Name: kw2docs docs2recs docs2kw titles urls
## Class: environment environment environment character character
##
## Name: doc_retriever
## Class: function
docset_cancer68
is the result of using the ssrch
function parseDoc
in conjunction with the CSV files zipped in ssrch/inst/cancer_corpus
.
It is an S4 class instance that manages environments mapping
from keywords to documents, documents to records, documents to keywords,
Furthermore, DocSet
instances can have a component that
retrieves parsed documents from the corpus.
We’ll search for the phrase Non Small Cell
with an optional hyphen,
ignoring character case of possible hits, including as well
the abbreviation for Non-small cell lung cancer.
## hits docs
## 1 NSCLC SRP093349
## 2 Non Small Cell Lung Cancer ERP013260
## 3 Non-Small Cell Lung Cancer (NSCLC) SRP107198
Three studies, three different patterns matched by the query string.
We can use retrieve_doc
to look at the contents
of the metadata tables for these studies.
The ctxsearch
function starts a shiny app that
provides access to full attribute data via a selectize
input that uses all tokens in the corpus (filtered).