Introduction

scPipe is a package designed to process single-cell RNA-sequencing (scRNA-seq) data generated by different protocols. It is designed for protocols with UMI, but can also adapt to non-UMI protocols. scPipe consist of two major components. The first is data preprocessing with raw fastq as input and a gene count matrix as output. The second component starts from the gene count matrix, includes quality control and a shiny app for clustering and visualization.

Case Study: Preprocessing CEL-seq2 data

CEL-seq2 is a modified version of the CEL-seq method with higher sensitivity and lower cost. It uses a polyT primer with index sequences to capture a cell’s mRNA. The index sequence consists of the a cell specific index, which is designed, and a molecular index, also called unique molecular identifier (UMI), which is a random sequence that should be different for each primer (the number of distinct UMIs will depend on its length). The ouput fastq files from a CEL-seq2 experiment is paired-ended where the first read contains the transcript and second read contains the cellindex+UMI with some polyA sequence.

Getting started

We begin by creating a folder to store the results.

library(scPipe)
library(SingleCellExperiment)
data_dir = tempdir()

Organising the files required

To process the data, we need the genome fasta file, gff3 exon annotation and a cell barcode annotation. The barcode annotation should be a .csv file with at least two columns, where the first column has the cell id and the second column contains the barcode sequence. We use ERCC spike-in genes for this demo. All files can be found in the extdata folder of the scPipe package:

# file path:
ERCCfa_fn = system.file("extdata", "ERCC92.fa", package = "scPipe")
ERCCanno_fn = system.file("extdata", "ERCC92_anno.gff3", package = "scPipe")
barcode_annotation_fn = system.file("extdata", "barcode_anno.csv", package = "scPipe")

The read structure for this example is paired-ended, with the first longer read mapping to transcripts and second shorter read consisting of 6bp UMIs followed by 8bp cell barcodes. NOTE: by convention, scPipe always assumes read1 refers to the read with the transcript sequence, which is usually the longer read. These data are also available in the in extdata folder:

fq_R1 = system.file("extdata", "simu_R1.fastq.gz", package = "scPipe")
fq_R2 = system.file("extdata", "simu_R2.fastq.gz", package = "scPipe")

Data Preprocessing

Fastq reformatting

The pipeline starts with fastq file reformatting. We move the barcode and UMI sequences to the read name and leave the transcript sequence as is. This outputs a read name that looks like @[barcode_sequence]*[UMI_sequence]#[readname] … The read structure of read 2 in our example dataset has the 8 bp long cell barcode starting at position 6 and the 6 bp long UMI sequence starting at the first position. So the read structure will be : list(bs1=-1, bl1=0, bs2=6, bl2=8, us=0, ul=6). bs1=-1, bl1=0 means we don’t have an index in read 1 so we set a negative value as its start position and give it zero length. bs2=6, bl2=8 means we have an index in read two which starts at position 6 in the read and is 8 bases in length. us=0, ul=6 means we have a UMI at the start of read 2 which is 6 bases long. NOTE: we use a zero based index system, so the indexing of the sequence starts at zero.

sc_trim_barcode(file.path(data_dir, "combined.fastq.gz"),
                fq_R1,
                fq_R2,
                read_structure = list(bs1=-1, bl1=0, bs2=6, bl2=8, us=0, ul=6))

## trimming fastq file...
## pass QC: 46009
## removed_have_N: 0
## removed_low_qual: 0
## time elapsed: 141 milliseconds

Aligning reads to a reference genome

Next we align reads to the genome. This example uses Rsubread but any aligner that support RNA-seq alignment and gives standard BAM output can be used here. The Rsubread package is not available on Windows, so the following alignment step will only run on Mac OSX or Linux.

if(.Platform$OS.type != "windows"){
  Rsubread::buildindex(basename=file.path(data_dir, "ERCC_index"), reference=ERCCfa_fn)

  Rsubread::align(index=file.path(data_dir, "ERCC_index"),
      readfile1=file.path(data_dir, "combined.fastq.gz"),
      output_file=file.path(data_dir, "out.aln.bam"), phredOffset=64)
}

## 
##         ==========     _____ _    _ ____  _____  ______          _____  
##         =====         / ____| |  | |  _ \|  __ \|  ____|   /\   |  __ \ 
##           =====      | (___ | |  | | |_) | |__) | |__     /  \  | |  | |
##             ====      \___ \| |  | |  _ <|  _  /|  __|   / /\ \ | |  | |
##               ====    ____) | |__| | |_) | | \ \| |____ / ____ \| |__| |
##         ==========   |_____/ \____/|____/|_|  \_\______/_/    \_\_____/
##        Rsubread 1.32.2
## 
## //=========================== indexBuilder setting ===========================\\
## ||                                                                            ||
## ||                Index name : ERCC_index                                     ||
## ||               Index space : base-space                                     ||
## ||           One block index : yes                                            ||
## ||          Repeat threshold : 100 repeats                                    ||
## ||  Distance to next subread : 1                                              ||
## ||                                                                            ||
## ||               Input files : 1 file in total                                ||
## ||                             o ERCC92.fa                                    ||
## ||                                                                            ||
## \\===================== http://subread.sourceforge.net/ ======================//
## 
## //================================= Running ==================================\\
## ||                                                                            ||
## || Check the integrity of provided reference sequences ...                    ||
## || There were 1 notes for reference sequences.                                ||
## || The notes can be found in the log file, '/tmp/RtmpeTveTv/ERCC_index.log'.  ||
## || Scan uninformative subreads in reference sequences ...                     ||
## ||    8%,   0 mins elapsed, rate=91.1k bps/s, total=0m                        ||
## ||   16%,   0 mins elapsed, rate=152.5k bps/s, total=0m                       ||
## ||   24%,   0 mins elapsed, rate=196.7k bps/s, total=0m                       ||
## ||   33%,   0 mins elapsed, rate=233.9k bps/s, total=0m                       ||
## ||   41%,   0 mins elapsed, rate=261.8k bps/s, total=0m                       ||
## ||   49%,   0 mins elapsed, rate=286.4k bps/s, total=0m                       ||
## ||   58%,   0 mins elapsed, rate=307.0k bps/s, total=0m                       ||
## ||   66%,   0 mins elapsed, rate=324.4k bps/s, total=0m                       ||
## ||   74%,   0 mins elapsed, rate=337.7k bps/s, total=0m                       ||
## ||   83%,   0 mins elapsed, rate=352.6k bps/s, total=0m                       ||
## ||   91%,   0 mins elapsed, rate=365.8k bps/s, total=0m                       ||
## || 1 uninformative subreads were found.                                       ||
## || These subreads were excluded from index building.                          ||
## || Build the index...                                                         ||
## ||   24%,   0 mins elapsed, rate=5262.1k bps/s, total=0m                      ||
## ||   49%,   0 mins elapsed, rate=5262.1k bps/s, total=0m                      ||
## ||  149%,   0 mins elapsed, rate=5740.4k bps/s, total=0m                      ||
## ||  183%,   0 mins elapsed, rate=5936.6k bps/s, total=0m                      ||
## ||  324%,   0 mins elapsed, rate=6080.6k bps/s, total=0m                      ||
## || Save current index block...                                                ||
## ||  [ 0.0% finished ]                                                         ||
## ||  [ 10.0% finished ]                                                        ||
## ||  [ 20.0% finished ]                                                        ||
## ||  [ 30.0% finished ]                                                        ||
## ||  [ 40.0% finished ]                                                        ||
## ||  [ 50.0% finished ]                                                        ||
## ||  [ 60.0% finished ]                                                        ||
## ||  [ 70.0% finished ]                                                        ||
## ||  [ 80.0% finished ]                                                        ||
## ||  [ 90.0% finished ]                                                        ||
## ||  [ 100.0% finished ]                                                       ||
## ||                                                                            ||
## ||                      Total running time: 0.0 minutes.                      ||
## ||          Index /tmp/RtmpeTveTv/ERCC_index was successfully built!          ||
## ||                                                                            ||
## \\===================== http://subread.sourceforge.net/ ======================//
## 
## 
##         ==========     _____ _    _ ____  _____  ______          _____  
##         =====         / ____| |  | |  _ \|  __ \|  ____|   /\   |  __ \ 
##           =====      | (___ | |  | | |_) | |__) | |__     /  \  | |  | |
##             ====      \___ \| |  | |  _ <|  _  /|  __|   / /\ \ | |  | |
##               ====    ____) | |__| | |_) | | \ \| |____ / ____ \| |__| |
##         ==========   |_____/ \____/|____/|_|  \_\______/_/    \_\_____/
##        Rsubread 1.32.2
## 
## //========================== subread-align setting ===========================\\
## ||                                                                            ||
## || Function      : Read alignment (RNA-Seq)                                   ||
## || Input file    : combined.fastq.gz                                          ||
## || Output file   : out.aln.bam (BAM)                                          ||
## || Index name    : ERCC_index                                                 ||
## ||                                                                            ||
## ||                    ------------------------------------                    ||
## ||                                                                            ||
## ||                               Threads : 1                                  ||
## ||                          Phred offset : 64                                 ||
## ||                             Min votes : 3 / 10                             ||
## ||                        Max mismatches : 3                                  ||
## ||                      Max indel length : 5                                  ||
## ||            Report multi-mapping reads : yes                                ||
## || Max alignments per multi-mapping read : 1                                  ||
## ||                                                                            ||
## \\===================== http://subread.sourceforge.net/ ======================//
## 
## //================ Running (04-Jan-2019 21:59:12, pid=17684) =================\\
## ||                                                                            ||
## || The input file contains base space reads.                                  ||
## || The range of Phred scores observed in the data is [1,1]                    ||
## || Load the 1-th index block...                                               ||
## ||   23% completed, 0.0 mins elapsed, rate=136.1k reads per second            ||
## ||   56% completed, 0.0 mins elapsed, rate=148.1k reads per second            ||
## ||   26% completed, 0.0 mins elapsed, rate=152.6k reads per second            ||
## ||   67% completed, 0.0 mins elapsed, rate=162.0k reads per second            ||
## ||   40% completed, 0.0 mins elapsed, rate=164.6k reads per second            ||
## ||   77% completed, 0.0 mins elapsed, rate=170.6k reads per second            ||
## ||   52% completed, 0.0 mins elapsed, rate=171.8k reads per second            ||
## ||   41% completed, 0.0 mins elapsed, rate=175.6k reads per second            ||
## ||   93% completed, 0.0 mins elapsed, rate=180.2k reads per second            ||
## ||   72% completed, 0.0 mins elapsed, rate=182.5k reads per second            ||
## ||   62% completed, 0.0 mins elapsed, rate=184.9k reads per second            ||
## ||   62% completed, 0.0 mins elapsed, rate=186.9k reads per second            ||
## ||   72% completed, 0.0 mins elapsed, rate=23.3k reads per second             ||
## ||   78% completed, 0.0 mins elapsed, rate=24.7k reads per second             ||
## ||   83% completed, 0.0 mins elapsed, rate=25.8k reads per second             ||
## ||   88% completed, 0.0 mins elapsed, rate=26.6k reads per second             ||
## ||   92% completed, 0.0 mins elapsed, rate=27.4k reads per second             ||
## ||   97% completed, 0.0 mins elapsed, rate=28.3k reads per second             ||
## ||  102% completed, 0.0 mins elapsed, rate=29.0k reads per second             ||
## ||  107% completed, 0.0 mins elapsed, rate=29.9k reads per second             ||
## ||  113% completed, 0.0 mins elapsed, rate=30.7k reads per second             ||
## ||                                                                            ||
## ||                          Completed successfully.                           ||
## ||                                                                            ||
## \\============================================================================//
## 
## //================================= Summary ==================================\\
## ||                                                                            ||
## ||         Total reads : 46,009                                               ||
## ||              Mapped : 46,009 (100.0%)                                      ||
## ||     Uniquely mapped : 46,009                                               ||
## ||       Multi-mapping : 0                                                    ||
## ||                                                                            ||
## ||            Unmapped : 0                                                    ||
## ||                                                                            ||
## ||              Indels : 0                                                    ||
## ||                                                                            ||
## ||        Running time : 0.0 minutes                                          ||
## ||                                                                            ||
## \\===================== http://subread.sourceforge.net/ ======================//

##                Category /tmp/RtmpeTveTv/out.aln.bam
## 1           Total_reads                       46009
## 2          Mapped_reads                       46009
## 3 Uniquely_mapped_reads                       46009
## 4   Multi_mapping_reads                           0
## 5        Unmapped_reads                           0
## 6                Indels                           0

Assigning reads to annotated exons

After the read alignment, we assign reads to exons based on the annotation provided using the sc_exon_mapping function. Currently scPipe only supports annotation in gff3 or bed format.

if(.Platform$OS.type != "windows"){
  sc_exon_mapping(file.path(data_dir, "out.aln.bam"),
                file.path(data_dir, "out.map.bam"),
                ERCCanno_fn)
}

## adding annotation files...
## time elapsed: 0 milliseconds
## 
## annotating exon features...
## updating progress every 3 minutes...
## 46009 reads processed, 46k reads/sec
## number of read processed: 46009
## unique map to exon: 46009 (100.00%)
## ambiguous map to multiple exon: 0 (0.00%)
## map to intron: 0 (0.00%)
## not mapped: 0 (0.00%)
## unaligned: 0 (0.00%)

De-multiplexing data and counting genes

Next we use the sc_demultiplex function to split the reads into separate .csv files for each cell in the /count subfolder. Each file contains three columns and each row corresponds to a distinct read. The first column contains the gene ID that the read maps to, the second column gives the UMI sequence for that read and the third column is the distance to the transcript end position (TES) in bp. This file will be used for UMI deduplication and to generate a gene count matrix by calling the sc_gene_counting function.

if(.Platform$OS.type != "windows"){
  sc_demultiplex(file.path(data_dir, "out.map.bam"), data_dir, barcode_annotation_fn,has_UMI=FALSE)

  sc_gene_counting(data_dir, barcode_annotation_fn)
}

## demultiplexing reads by barcode...
## Warning: mitochondrial chromosome not found using chromosome name `MT`.
## time elapsed: 107 milliseconds
## 
## summarising gene counts...
## time elapsed: 96 milliseconds

We have now completed the preprocessing steps. The gene count matrix is available as a .csv file in data_dir/gene_count.csv and quality control statistics are saved in the data_dir/stat folder. These data are useful for later quality control (QC).

Preprocessing data generated by other protocols

The greatest difference between scRNA-seq protocols is in their read structure, which is specified by the read_structure argument of the sc_trim_barcode function in scPipe. The major difference between CEL-seq and Drop-seq/Chromium 10x is that the cell barcode is unknown in advance for the latter protocols, so an extra step sc_detect_bc is required before running sc_demultiplex, to identify and generate the cell barcode annotation. scPipe is not optimized for the full-lengh protocols like SMART-seq, but it can process the data generated by such protocols by setting the has_UMI to FALSE in the sc_demultiplex function.

Quality Control

The easiest way to create a SingleCellExperiment object from the output of scPipe preprocessing is using create_sce_by_dir function, that will read in the gene count matrix together with the QC information available in the stat folder.

if(.Platform$OS.type != "windows"){
sce = create_sce_by_dir(data_dir)
dim(sce)
}

## [1] 92 10

The dataset we analysed above used ERCC simulated reads from 10 cells of perfect quality. In order to demonstrate QC on a more reaslitic example, we have included an experimental dataset in the scPipe software generated by Dr Christine Biben. This experiment profiles gene expression in 383 blood cells for a subset of 1000 genes (to save space). We first create a SingleCellExperiment object directly without using the create_sce_by_dir function:

data("sc_sample_data")
data("sc_sample_qc")
sce = SingleCellExperiment(assays = list(counts = as.matrix(sc_sample_data))) # generate new sce with gene count matrix
QC_metrics(sce) = sc_sample_qc
demultiplex_info(sce) = cell_barcode_matching

## organism/gene_id_type not provided. Make a guess: mmusculus_gene_ensembl / ensembl_gene_id

UMI_dup_info(sce) = UMI_duplication

There are several plots we can create to assess the overall quality of the experiment. The first is to look at the cell barcode demultiplexing statistics. To do this we generate a bar plot that shows the percentage of reads that uniquely match to the cell barcodes, as well as the unmatched proportion of reads and their alignment rates to introns and exons. If we observe a large proportion of unmatched reads that map to exons, this indicates a failure of the cell barcode demultiplexing.

plot_demultiplex(sce)

A second plot shows the duplication rate which can be used to evaluate read deapth. UMIs are routinely used to mark individual molecules and after PCR amplification, the same molecule with have multiple copys, which can be identified and removed if they are observed to have the same UMI sequence. Therefore, the copy number of UMIs is an indication of the PCR amplification rate.

plot_UMI_dup(sce)

Next we calculate QC metrics and use the detect_outlier function to identify poor quality cells. The detect_outlier function has argument comp to define the maximum component of the gaussian mixture model. Using the default value of 1 is generally sufficient, but in cases where the data are heterogeneous in terms of quality control metrics, setting this value to 2 or 3 can give better results. This function will remove low quality cells if type="low", large cells if type="high" or both when type="both". The conf argument specifies the lower and upper confidence intervals for outliers and detect_outlier is insensistive to the interval values.

sce = calculate_QC_metrics(sce)
sce = detect_outlier(sce)

We can plot the alignment statistics per sample as a barplot using

plot_mapping(sce, percentage = TRUE, dataname = "sc_sample_data")

and generate pairwise plots of the QC metrics as follows:

plot_QC_pairs(sce)

The final step will be to remove the low quality cells by the remove_outliers function.

sce = remove_outliers(sce)
dim(sce)

## [1] 1000  362

Downstream analysis

Since the scater and scran packages both use the SingleCellExperiment class, it will be easy to further process this data using these packages for normalization and visualization. Other packages such as SC3 may be useful for clustering and MAST and edgeR for differential expression analysis.

sessionInfo()

## R version 3.5.2 (2018-12-20)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 16.04.5 LTS
## 
## Matrix products: default
## BLAS: /home/biocbuild/bbs-3.8-bioc/R/lib/libRblas.so
## LAPACK: /home/biocbuild/bbs-3.8-bioc/R/lib/libRlapack.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=en_US.UTF-8    
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=C              
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] parallel  stats4    stats     graphics  grDevices utils     datasets 
## [8] methods   base     
## 
## other attached packages:
##  [1] bindrcpp_0.2.2              scPipe_1.4.1               
##  [3] SingleCellExperiment_1.4.1  SummarizedExperiment_1.12.0
##  [5] DelayedArray_0.8.0          BiocParallel_1.16.5        
##  [7] matrixStats_0.54.0          Biobase_2.42.0             
##  [9] GenomicRanges_1.34.0        GenomeInfoDb_1.18.1        
## [11] IRanges_2.16.0              S4Vectors_0.20.1           
## [13] BiocGenerics_0.28.0         ggplot2_3.1.0              
## 
## loaded via a namespace (and not attached):
##  [1] httr_1.4.0               bit64_0.9-7             
##  [3] assertthat_0.2.0         blob_1.1.1              
##  [5] Rsamtools_1.34.0         GenomeInfoDbData_1.2.0  
##  [7] yaml_2.2.0               robustbase_0.93-3       
##  [9] progress_1.2.0           pillar_1.3.1            
## [11] RSQLite_2.1.1            lattice_0.20-38         
## [13] glue_1.3.0               digest_0.6.18           
## [15] RColorBrewer_1.1-2       XVector_0.22.0          
## [17] colorspace_1.3-2         htmltools_0.3.6         
## [19] Matrix_1.2-15            plyr_1.8.4              
## [21] XML_3.98-1.16            pkgconfig_2.0.2         
## [23] biomaRt_2.38.0           hashmap_0.2.2           
## [25] zlibbioc_1.28.0          org.Mm.eg.db_3.7.0      
## [27] purrr_0.2.5              scales_1.0.0            
## [29] tibble_2.0.0             withr_2.1.2             
## [31] lazyeval_0.2.1           magrittr_1.5            
## [33] crayon_1.3.4             mclust_5.4.2            
## [35] memoise_1.1.0            evaluate_0.12           
## [37] GGally_1.4.0             MASS_7.3-51.1           
## [39] tools_3.5.2              prettyunits_1.0.2       
## [41] hms_0.4.2                org.Hs.eg.db_3.7.0      
## [43] stringr_1.3.1            Rhtslib_1.14.0          
## [45] munsell_0.5.0            Biostrings_2.50.2       
## [47] AnnotationDbi_1.44.0     compiler_3.5.2          
## [49] rlang_0.3.0.1            grid_3.5.2              
## [51] RCurl_1.95-4.11          labeling_0.3            
## [53] bitops_1.0-6             rmarkdown_1.11          
## [55] Rsubread_1.32.2          codetools_0.2-16        
## [57] gtable_0.2.0             curl_3.2                
## [59] DBI_1.0.0                reshape_0.8.8           
## [61] reshape2_1.4.3           R6_2.3.0                
## [63] GenomicAlignments_1.18.1 rtracklayer_1.42.1      
## [65] knitr_1.21               dplyr_0.7.8             
## [67] bit_1.1-14               bindr_0.1.1             
## [69] stringi_1.2.4            Rcpp_1.0.0              
## [71] DEoptimR_1.0-8           tidyselect_0.2.5        
## [73] xfun_0.4

scPipe: a flexible data preprocessing pipeline for 3’ end scRNA-seq data

Luyi Tian

2019-01-04