# Saving and reloading DelayedArray objects

# Motivation

The chihaya package saves `DelayedArray` objects for efficient, portable and stable reproduction of delayed operations in a new R session or other programming frameworks.

- Portability. We provide a file specification in a standard format (HDF5) that enables other languages to easily interpret and reproduce the delayed operations.
- Stability. By converting DelayedArray operations into our specification, we provide a layer of protection against changes in the S4 class structure that would invalidate serialized RDS objects.
- Efficiency. We avoid any realization of the delayed operations, enabling quick saving and avoiding loss of data structure in the seed (e.g., sparsity).

Check out the specification for more details.

# Quick start

Make a `DelayedArray` object with some operations:

    library(DelayedArray)
    x <- DelayedArray(matrix(runif(1000), ncol=10))
    x <- x[11:15,] / runif(5)
    x <- log2(x + 1)
    x

    ## <5 x 10> DelayedMatrix object of type "double":
    ##            [,1]       [,2]       [,3] ...      [,9]     [,10]
    ## [1,] 1.08548934 0.67414925 0.96349128   . 0.3276063 0.9043479
    ## [2,] 3.04063274 0.83612516 2.63562792   . 2.9714434 2.0214361
    ## [3,] 0.29797166 0.29997230 0.95836369   . 0.1555044 0.1214245
    ## [4,] 1.95815989 2.79877300 2.35532559   . 2.5798019 2.0613630
    ## [5,] 2.11391534 0.03981024 2.91688987   . 2.7249142 1.6300176

    showtree(x)

    ## 5x10 double: DelayedMatrix object
    ## └─ 5x10 double: Stack of 2 unary iso op(s)
    ##    └─ 5x10 double: Unary iso op with args
    ##       └─ 5x10 double: Subset
    ##          └─ 100x10 double: [seed] matrix object

Save it into a HDF5 file with `saveDelayed()`:

    library(chihaya)
    tmp <- tempfile(fileext=".h5")
    saveDelayed(x, tmp)
    rhdf5::h5ls(tmp)

    ##                            group    name       otype  dclass      dim
    ## 0                              / delayed   H5I_GROUP
    ## 1                       /delayed    base H5I_DATASET   FLOAT    ( 0 )
    ## 2                       /delayed  method H5I_DATASET  STRING    ( 0 )
    ## 3                       /delayed    seed   H5I_GROUP
    ## 4                  /delayed/seed  method H5I_DATASET  STRING    ( 0 )
    ## 5                  /delayed/seed    seed   H5I_GROUP
    ## 6             /delayed/seed/seed   along H5I_DATASET INTEGER    ( 0 )
    ## 7             /delayed/seed/seed  method H5I_DATASET  STRING    ( 0 )
    ## 8             /delayed/seed/seed    seed   H5I_GROUP
    ## 9        /delayed/seed/seed/seed   index   H5I_GROUP
    ## 10 /delayed/seed/seed/seed/index       0 H5I_DATASET INTEGER        5
    ## 11       /delayed/seed/seed/seed    seed   H5I_GROUP
    ## 12  /delayed/seed/seed/seed/seed    data H5I_DATASET   FLOAT 100 x 10
    ## 13  /delayed/seed/seed/seed/seed  native H5I_DATASET INTEGER    ( 0 )
    ## 14            /delayed/seed/seed    side H5I_DATASET  STRING    ( 0 )
    ## 15            /delayed/seed/seed   value H5I_DATASET   FLOAT        5
    ## 16                 /delayed/seed    side H5I_DATASET  STRING    ( 0 )
    ## 17                 /delayed/seed   value H5I_DATASET   FLOAT    ( 0 )

And then load it back in later:

    y <- loadDelayed(tmp)
    y

    ## <5 x 10> DelayedMatrix object of type "double":
    ##            [,1]       [,2]       [,3] ...      [,9]     [,10]
    ## [1,] 1.08548934 0.67414925 0.96349128   . 0.3276063 0.9043479
    ## [2,] 3.04063274 0.83612516 2.63562792   . 2.9714434 2.0214361
    ## [3,] 0.29797166 0.29997230 0.95836369   . 0.1555044 0.1214245
    ## [4,] 1.95815989 2.79877300 2.35532559   . 2.5798019 2.0613630
    ## [5,] 2.11391534 0.03981024 2.91688987   . 2.7249142 1.6300176

Of course, this is not a particularly interesting case as we end up saving the original array inside our HDF5 file anyway. The real fun begins when you have some more interesting seeds.

# More interesting seeds

We can use the delayed nature of the operations to avoid breaking sparsity. For example:

    library(Matrix)
    x <- rsparsematrix(1000, 1000, density=0.01)
    x <- DelayedArray(x) + runif(1000)

    tmp <- tempfile(fileext=".h5")
    saveDelayed(x, tmp)
    rhdf5::h5ls(tmp)

    ##            group     name       otype  dclass   dim
    ## 0              /  delayed   H5I_GROUP
    ## 1       /delayed    along H5I_DATASET INTEGER ( 0 )
    ## 2       /delayed   method H5I_DATASET  STRING ( 0 )
    ## 3       /delayed     seed   H5I_GROUP
    ## 4  /delayed/seed     data H5I_DATASET   FLOAT 10000
    ## 5  /delayed/seed dimnames   H5I_GROUP
    ## 6  /delayed/seed  indices H5I_DATASET INTEGER 10000
    ## 7  /delayed/seed   indptr H5I_DATASET INTEGER  1001
    ## 8  /delayed/seed    shape H5I_DATASET INTEGER     2
    ## 9       /delayed     side H5I_DATASET  STRING ( 0 )
    ## 10      /delayed    value H5I_DATASET   FLOAT  1000

    file.info(tmp)[["size"]]

    ## [1] 101923

    # Compared to a dense array.
    tmp2 <- tempfile(fileext=".h5")
    out <- HDF5Array::writeHDF5Array(x, tmp2, "data")
    file.info(tmp2)[["size"]]

    ## [1] 280197

    # Loading it back in.
    y <- loadDelayed(tmp)
    showtree(y)

    ## 1000x1000 double: DelayedMatrix object
    ## └─ 1000x1000 double: Unary iso op with args
    ##    └─ 1000x1000 double, sparse: [seed] dgCMatrix object

We can also store references to external files, thus avoiding data duplication:

    library(HDF5Array)
    test <- HDF5Array(tmp2, "data")
    stuff <- log2(test + 1)
    stuff

    ## <1000 x 1000> DelayedMatrix object of type "double":
    ##                [,1]        [,2]        [,3] ...      [,999]     [,1000]
    ##    [1,]   0.7967278   0.7967278   0.7967278   .   0.7967278   0.7967278
    ##    [2,]   0.4854093   0.4854093   0.4854093   .   0.4854093   0.4854093
    ##    [3,]   0.7950080   0.7950080   0.7950080   .   0.7950080   0.7950080
    ##    [4,]   0.6716864   0.6716864   0.6716864   .   0.6716864   0.6716864
    ##    [5,]   0.4516181   0.4516181   0.4516181   .   0.4516181   0.4516181
    ##     ...           .           .           .   .           .           .
    ##  [996,] 0.003586764 0.003586764 0.003586764   . 0.003586764 0.003586764
    ##  [997,] 0.846503190 0.846503190 0.846503190   . 0.846503190 0.846503190
    ##  [998,] 0.249288363 0.249288363 0.249288363   . 0.249288363 0.249288363
    ##  [999,] 0.203928021 0.203928021 0.203928021   . 0.203928021 0.203928021
    ## [1000,] 0.342281848 0.342281848 0.342281848   . 0.342281848 0.342281848

    tmp <- tempfile(fileext=".h5")
    saveDelayed(stuff, tmp)
    rhdf5::h5ls(tmp)

    ##                 group       name       otype  dclass   dim
    ## 0                   /    delayed   H5I_GROUP
    ## 1            /delayed       base H5I_DATASET   FLOAT ( 0 )
    ## 2            /delayed     method H5I_DATASET  STRING ( 0 )
    ## 3            /delayed       seed   H5I_GROUP
    ## 4       /delayed/seed     method H5I_DATASET  STRING ( 0 )
    ## 5       /delayed/seed       seed   H5I_GROUP
    ## 6  /delayed/seed/seed dimensions H5I_DATASET INTEGER     2
    ## 7  /delayed/seed/seed       file H5I_DATASET  STRING ( 0 )
    ## 8  /delayed/seed/seed       name H5I_DATASET  STRING ( 0 )
    ## 9  /delayed/seed/seed     sparse H5I_DATASET INTEGER ( 0 )
    ## 10 /delayed/seed/seed       type H5I_DATASET  STRING ( 0 )
    ## 11      /delayed/seed       side H5I_DATASET  STRING ( 0 )
    ## 12      /delayed/seed      value H5I_DATASET   FLOAT ( 0 )

    file.info(tmp)[["size"]] # size of the delayed operations + pointer to the actual file

    ## [1] 49642

    y <- loadDelayed(tmp)
    y

    ## <1000 x 1000> DelayedMatrix object of type "double":
    ##                [,1]        [,2]        [,3] ...      [,999]     [,1000]
    ##    [1,]   0.7967278   0.7967278   0.7967278   .   0.7967278   0.7967278
    ##    [2,]   0.4854093   0.4854093   0.4854093   .   0.4854093   0.4854093
    ##    [3,]   0.7950080   0.7950080   0.7950080   .   0.7950080   0.7950080
    ##    [4,]   0.6716864   0.6716864   0.6716864   .   0.6716864   0.6716864
    ##    [5,]   0.4516181   0.4516181   0.4516181   .   0.4516181   0.4516181
    ##     ...           .           .           .   .           .           .
    ##  [996,] 0.003586764 0.003586764 0.003586764   . 0.003586764 0.003586764
    ##  [997,] 0.846503190 0.846503190 0.846503190   . 0.846503190 0.846503190
    ##  [998,] 0.249288363 0.249288363 0.249288363   . 0.249288363 0.249288363
    ##  [999,] 0.203928021 0.203928021 0.203928021   . 0.203928021 0.203928021
    ## [1000,] 0.342281848 0.342281848 0.342281848   . 0.342281848 0.342281848

# Session information

    sessionInfo()

    ## R version 4.5.1 (2025-06-13)
    ## Platform: x86_64-pc-linux-gnu
    ## Running under: Ubuntu 24.04.3 LTS
    ##
    ## Matrix products: default
    ## BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
    ## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0
    ##
    ## locale:
    ##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
    ##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=C
    ##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
    ##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
    ##  [9] LC_ADDRESS=C               LC_TELEPHONE=C
    ## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
    ##
    ## time zone: Etc/UTC
    ## tzcode source: system (glibc)
    ##
    ## attached base packages:
    ## [1] stats4    stats     graphics  grDevices utils     datasets  methods
    ## [8] base
    ##
    ## other attached packages:
    ##  [1] HDF5Array_1.37.0      h5mread_1.1.1         rhdf5_2.53.6
    ##  [4] chihaya_1.11.0        DelayedArray_0.35.5   SparseArray_1.9.2
    ##  [7] S4Arrays_1.9.3        abind_1.4-8           IRanges_2.43.8
    ## [10] S4Vectors_0.47.6      MatrixGenerics_1.21.0 matrixStats_1.5.0
    ## [13] BiocGenerics_0.55.4   generics_0.1.4        Matrix_1.7-4
    ## [16] BiocStyle_2.37.1
    ##
    ## loaded via a namespace (and not attached):
    ##  [1] jsonlite_2.0.0      compiler_4.5.1      BiocManager_1.30.26
    ##  [4] Rcpp_1.1.0          rhdf5filters_1.21.4 jquerylib_0.1.4
    ##  [7] yaml_2.3.10         fastmap_1.2.0       lattice_0.22-7
    ## [10] R6_2.6.1            XVector_0.49.3      knitr_1.50
    ## [13] maketools_1.3.2     bslib_0.9.0         rlang_1.1.6
    ## [16] cachem_1.1.0        xfun_0.53           sass_0.4.10
    ## [19] sys_3.4.3           cli_3.6.5           Rhdf5lib_1.31.1
    ## [22] digest_0.6.37       grid_4.5.1          lifecycle_1.0.4
    ## [25] evaluate_1.0.5      buildtools_1.0.0    rmarkdown_2.30
    ## [28] tools_4.5.1         htmltools_0.5.8.1