Contents

1 Overview

beachmat has a few useful utilities outside of the C++ API. This document describes how to use them.

2 Choosing HDF5 chunk dimensions

Given the dimensions of a matrix, users can choose HDF5 chunk dimensions that give fast performance for both row- and column-level access.

library(beachmat)
nrows <- 10000
ncols <- 200
getBestChunkDims(c(nrows, ncols))
## [1] 708  15

In the future, it should be possible to feed this back into the API. Currently, if chunk dimensions are not specified in the C++ code, the API will retrieve them from R via the getHDF5DumpChunkDim() function from HDF5Array. The aim is to also provide a setHDF5DumpChunkDim() function so that any chunk dimension specified in R will be respected.

3 Rechunking a HDF5 file

The most common access patterns for matrices (at least, for high-throughput biological data) is by row or by column. The rechunkByMargins() will take a HDF5 file and convert it to using purely row- or column-based chunks.

library(HDF5Array)
A <- as(matrix(runif(5000), nrow=100, ncol=50), "HDF5Array")
byrow <- rechunkByMargins(A, byrow=TRUE)
byrow
## <100 x 50> HDF5Matrix object of type "double":
##               [,1]        [,2]        [,3] ...      [,49]      [,50]
##   [1,]  0.02368997  0.61867999  0.09901986   . 0.94371845 0.20936938
##   [2,]  0.16157399  0.08843818  0.38171039   . 0.47664483 0.01198130
##   [3,]  0.31990193  0.79708877  0.95708924   . 0.01796980 0.33201111
##   [4,]  0.21734044  0.66196205  0.86554829   . 0.65668447 0.05624321
##   [5,]  0.20153414  0.27811737  0.11535485   . 0.71071996 0.91199576
##    ...           .           .           .   .          .          .
##  [96,] 0.764048594 0.529289348 0.938423564   . 0.45495960 0.77103545
##  [97,] 0.096762599 0.144585831 0.001744258   . 0.77428899 0.86295320
##  [98,] 0.130667802 0.556895541 0.983594731   . 0.73824354 0.82954800
##  [99,] 0.844224918 0.866666283 0.643571294   . 0.20662298 0.02184707
## [100,] 0.182871031 0.985669492 0.982775865   . 0.17300827 0.88982778
bycol <- rechunkByMargins(A, byrow=FALSE)
bycol
## <100 x 50> HDF5Matrix object of type "double":
##               [,1]        [,2]        [,3] ...      [,49]      [,50]
##   [1,]  0.02368997  0.61867999  0.09901986   . 0.94371845 0.20936938
##   [2,]  0.16157399  0.08843818  0.38171039   . 0.47664483 0.01198130
##   [3,]  0.31990193  0.79708877  0.95708924   . 0.01796980 0.33201111
##   [4,]  0.21734044  0.66196205  0.86554829   . 0.65668447 0.05624321
##   [5,]  0.20153414  0.27811737  0.11535485   . 0.71071996 0.91199576
##    ...           .           .           .   .          .          .
##  [96,] 0.764048594 0.529289348 0.938423564   . 0.45495960 0.77103545
##  [97,] 0.096762599 0.144585831 0.001744258   . 0.77428899 0.86295320
##  [98,] 0.130667802 0.556895541 0.983594731   . 0.73824354 0.82954800
##  [99,] 0.844224918 0.866666283 0.643571294   . 0.20662298 0.02184707
## [100,] 0.182871031 0.985669492 0.982775865   . 0.17300827 0.88982778

Rechunking can provide a substantial speed-up to downstream functions, especially those requiring access to random columns or rows. Indeed, the time saved in those functions often offsets the time spent in constructing a new HDF5Matrix.

4 Session information

sessionInfo()
## R version 3.5.1 Patched (2018-07-12 r74967)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 16.04.5 LTS
## 
## Matrix products: default
## BLAS: /home/biocbuild/bbs-3.8-bioc/R/lib/libRblas.so
## LAPACK: /home/biocbuild/bbs-3.8-bioc/R/lib/libRlapack.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=C              
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] parallel  stats4    stats     graphics  grDevices utils     datasets 
## [8] methods   base     
## 
## other attached packages:
##  [1] HDF5Array_1.10.0    rhdf5_2.26.0        DelayedArray_0.8.0 
##  [4] BiocParallel_1.16.0 IRanges_2.16.0      S4Vectors_0.20.0   
##  [7] BiocGenerics_0.28.0 matrixStats_0.54.0  beachmat_1.4.0     
## [10] knitr_1.20          BiocStyle_2.10.0   
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_0.12.19       magrittr_1.5       stringr_1.3.1     
##  [4] tools_3.5.1        xfun_0.4           htmltools_0.3.6   
##  [7] yaml_2.2.0         rprojroot_1.3-2    digest_0.6.18     
## [10] bookdown_0.7       Rhdf5lib_1.4.0     BiocManager_1.30.3
## [13] evaluate_0.12      rmarkdown_1.10     stringi_1.2.4     
## [16] compiler_3.5.1     backports_1.1.2