Changes in version 0.99.34 Bug fixes - writeParquet() of a MultiAssaySpatialExperiment with an image or label that is not a file or a label matrix wrote only a metadata.json holding the name and type, with no path and no pixels, and readParquet() then failed on the whole package with "RasterLayerList ... must have dim()". Images and labels are stored as references, so a package must hold one that readParquet() can follow or refuse to be written: - A ZarrArray is now written as a reference to its store, which stays where it is, and read back as a ZarrArray. - An image held in memory, which has no file or store to refer to, is now an error at write time, and the message says to save it as an image file (a StoredSpatialImage) or as a Zarr store. A StoredSpatialImage whose file no longer exists is an error too, where before it was written without a path. A label matrix is unchanged: it is stored as coordinates. - A reference that cannot be followed when reading, because the file or the store has gone or ZarrArray is not installed, now leaves that image or label out with a warning that names it, instead of failing the package. - The branch that wrote empty scale_i directories for a list of ZarrArrays is gone: a RasterLayerList cannot hold such a list, so it could not be reached. Changes in version 0.99.33 Bug fixes - writeParquet()'s arrow::write_parquet()/write_dataset() calls hardcoded compression = "zstd", which errors on a "minimal" arrow build without zstd codec support (see DuckDBDataFrame#1). Now uses DuckDBDataFrame::resolveParquetCompression(), which falls back to snappy (with no compression level, since a level paired with a non-zstd codec errors on both DuckDB and arrow) when zstd is unavailable, and still prefers zstd otherwise. Changes in version 0.99.32 Bug fixes - Correctness: weighted colPair/rowPair graphs (graph_edges layout) silently lost their weight column on round trip, in both directions. .schema_datacols() excluded every value listed under the graphEdges schema key from the columns it keeps as mcols -- including weight, which (unlike from/to) is not a structural column name. The reverse direction had the mirror gap: .addGraphMetadata() never recorded a weight role when writing a SelfHits/DuckDBSelfHits with a weight mcol. Fixed on both the read side (.schema_datacols() now excludes only the structural from/to names) and the write side (both writeParquet methods -- SelfHits and the separate, lazy DuckDBSelfHits one -- now detect a weight mcol and record it). inst/schema/biocduckdb-profile.json's graphEdges block now documents the optional weight property. Changes in version 0.99.31 Bug fixes - Fixed two Bioconductor nebbiolo2 build-report NOTEs, plus the ERROR from DuckDBDataFrame's own undeclared nanoarrow dependency (see DuckDBDataFrame's NEWS) that was failing writeParquet()-based tests here. Vendored a local .sampleMapFromData() helper (used by .newMAE() to auto-derive a sampleMap when none is supplied) instead of calling unexported MultiAssayExperiment:::.sampleMapFromData(). Fixed a malformed Rd cross-reference (\link[BiocSingular]{BiocSingularParam}, which has no matching alias; the real target is BiocSingularParam-class) and replaced unanchored \linkS4class{DuckDBMatrix}/\linkS4class{DuckDBArraySeed} references (classes defined in DuckDBArray) with explicit \link[DuckDBArray]{...} package anchors. - Correctness: normalizeCounts() on DuckDBMatrix silently ignored size_factors/center_size_factors (the underscore-spelled arguments batchelor::fastMNN()'s cosine-normalization step actually calls it with), and computed library-size normalization instead of the requested explicit-size-factor normalization. scuttle's own methods accept both the historical dotted argument spellings (size.factors, center.size.factors, ...) and newer underscore spellings for the same arguments, with a non-NULL underscore value overriding its dotted counterpart; this method only ever declared the dotted formals, so a caller using the underscore spelling had its arguments silently fall into ... unused. Confirmed up to a 16.9 absolute difference against the correct result on a small fixture, and a full fastMNN() divergence on real data. Fixed by also accepting size_factors, pseudo_count, center_size_factors, and subset_row as aliases of their dotted counterparts. - The same gap existed in 8 other DuckDBMatrix methods wherever scuttle itself defines an underscore alias: calculateAverage()/calculateCPM()/ calculateTPM() (size_factors, subset_row), librarySizeFactors() (subset_row), numDetectedAcrossFeatures() (subset_row, subset_col, detection_limit, the last aliasing threshold, not just an underscore respelling), sumCountsAcrossFeatures() (subset_row, subset_col), perCellQCMetrics() (percent_top, detection_limit), and perFeatureQCMetrics() (detection_limit). geometricSizeFactors() and summarizeAssayByGroup() are unaffected: scuttle defines no underscore aliases for either. Verified by reading scuttle's own source for every alias it defines per generic, not by pattern-matching dotted-to-underscore (several aliases, like detection_limit/threshold and percent_top/percent.top, rename the argument outright and would not be caught by a mechanical substitution). Regression tests added for all 9 methods. Changes in version 0.99.30 Enhancements - modelGeneVar()/modelGeneVarByPoisson(), modelGeneCV2(), and pairwiseTTests()/findMarkers() on DuckDBMatrix no longer compute per-gene/per-block means and variances with two separate SQL passes (one for means, one that rejoined them to sum squared deviations). Both are now fused into a single GROUP BY query using DuckDB's native VAR_SAMP(); the sparse "fill" population is folded in afterward with a closed-form correction, so no raw sum-of-squares is ever computed. Verified bit-exact against the previous output, including an adversarial high-mean/low-variance gene. About 2.3-6.3x faster depending on call site. - perCellQCMetrics(), perFeatureQCMetrics(), numDetectedAcrossFeatures(), sumCountsAcrossFeatures(), and summarizeAssayByGroup() on DuckDBMatrix dropped a filter that ran immediately before an already-equivalent join (the filter's predicate and the join always select the same rows). This also avoids inlining a potentially million-entry IN (...) list into the generated SQL at this package's documented million-cell benchmark scale. Output is unchanged. About 1.2x faster at typical scale; negligible to no benefit at larger cell counts. - normalizeCounts()'s default path (also used by logNormCounts()), and the equivalent default paths of calculateAverage()/calculateCPM(), now compute library-size normalization in one query (two chained window functions) instead of collecting the per-cell size factors into R and rejoining them. Output is unchanged, including existing NA-propagation behavior. This simplifies the query plan but is not a speedup: measured 1.3-1.7x slower than the previous implementation across a range of matrix shapes, likely because of the window operator's own cost. Kept for the simpler, join-free query plan; if you rely on normalizeCounts() throughput, benchmark on your own data before upgrading. Bug fixes - Correctness: correlatePairs() on DuckDBMatrix could silently return Pearson correlations outside [-1, 1], or NaN, on non-log-normalized or large-magnitude input (raw counts, pseudobulk sums, proteomics/imaging intensities). The old formula computed correlation from raw sums and sums-of-squares, which loses precision catastrophically once values sit far from zero. If you saw rho values outside [-1, 1] or NaN, this is why. Replaced with a numerically stable, sparse-aware computation built on DuckDB's native VAR_SAMP()/COVAR_SAMP(); verified against R's own cor()/cov() across realistic and deliberately adversarial scales, with no regression on typical log-normalized data. About 15% slower than the old (incorrect) implementation. - .duckdb_seed_to_sparse_matrix()'s memory_limit guard underestimated its real transient memory cost (assumed 16 bytes/nonzero; measured 92-214 bytes/nonzero for string-keyed data, the common case when reading a COO Parquet file written by another tool). The guard could therefore let through a materialization that used several times more memory than the configured limit. Now uses a conservative 256 bytes/nonzero estimate. Changes in version 0.99.29 Bug fixes - .duckdb_seed_to_sparse_matrix() (the beachmat/tatami fast path behind initializeCpp,DuckDBArraySeed-method, used by BiocSingular::runSVD's compute_center/compute_scale for PCA) materialized the seed's raw, unprojected column instead of its logical (datacols-level) one. tblconn(table, select = FALSE) deliberately skips column-selection, so the connection it returns reflects the table's base columns, not whatever SQL projection (e.g. a log1p() transform applied via Math/Ops) is recorded in datacols; the subsequent select(conn, all_of(datacol_name)) then picked that raw column up by name, which silently succeeds whenever a transformed datacol keeps its original name (the common case: Math/Ops rewrite a datacol's expression in place, e.g. value to log(value + 1), without renaming it). Reported with a reproducible fixture showing up to a 0.405 error in PCA centering on a log-transformed assay versus the correctly-projected result (down to ~1.3e-15, floating-point noise, after the fix). Fixed by using tblconn(table, select = TRUE) (the generic's own default) so the same select() call afterward operates on the already-projected connection. Confirmed no existing test asserted the previous (buggy) behavior; added regression tests covering both the direct value-extraction path and an end-to-end runSVD/PCA comparison against a log1p-transformed oracle. Changes in version 0.99.28 Bug fixes - spatialJoin was missing the mixed-type method pair (c("DuckDBDataFrame", "DataFrame")/c("DataFrame", "DuckDBDataFrame")) that spatialMatch already has. A call with one out-of-core DuckDBDataFrame side and one in-memory DataFrame side (in either order) therefore had no applicable method more specific than MultiAssaySpatialExperiment's own c("DataFrame", "DataFrame") fallback, which materializes both sides' geometries fully into memory via st_as_sfc() -- silently defeating the out-of-core path for exactly the mixed case a caller joining a lazy layer against a small in-memory one would hit. Found via user feedback flagging that geometries were still loading into memory. Fixed by adding the missing method pair, delegating to DuckDBSpatial::st_join_table()/sf::st_join() the same way the existing c("DuckDBDataFrame", "DuckDBDataFrame") method does (both already accept a mixed lazy/in-memory pair as of DuckDBSpatial 0.99.8). Added a regression test covering both argument orders. - Same gap, one level deeper: spatialMatch/spatialJoin also had no method for a plain sf object (an sf is a base S3 class, c("sf", "data.frame"), not an S4Vectors::DataFrame, so the DataFrame- signature methods above never matched it; calling either with a real sf object errored with "unable to find an inherited method" rather than falling through to anything). This is purely a missing signature, not an S3/S4 dispatch limitation: setMethod(generic, c(..., "sf"), ...) dispatches on the class vector directly, no setOldClass() bridging needed. Fixed by adding c("DuckDBDataFrame", "sf") for spatialMatch, and both c("DuckDBDataFrame", "sf")/c("sf", "DuckDBDataFrame") for spatialJoin, delegating to the same existing DuckDBSpatial calls (which already handle sf input correctly as of DuckDBSpatial 0.99.8). Added regression tests for all three new signatures. Changes in version 0.99.27 Bug fixes - spatialJoin(DuckDBDataFrame, DuckDBDataFrame) with its default sparse = TRUE called DuckDBSpatial::st_intersects_table(x, y), which only ever accepted y as a single scalar geometry (sfc/sfg/WKT); it always errored with "unsupported geometry type: DuckDBDataFrame" when both sides were out-of-core, defeating the whole point of the sparse, stay-lazy path (the sparse = FALSE branch, a full attribute join via sf::st_join(), worked fine already). Fixed by switching to DuckDBSpatial::st_join_table() (new in DuckDBSpatial 0.99.8), a real two-table SQL join producing a sparse (id_x, id_y) pair table instead of embedding y as a literal. Changes in version 0.99.26 Bug fixes - The spatial_label_coord write path for a matrix/array-valued spatialLabels() entry (writeParquet,MultiAssaySpatialExperiment-method) was broken and had never been exercised by any test or fixture. The internal .matrixToCoordLabel() built a COO triples data.frame (row, col, value columns) and passed it to DuckDBArray::writeCoordArray() along with row=/col= arguments -- writeCoordArray() has no such parameters (they belong to the unrelated DuckDBMatrix() constructor), so they were silently swallowed by ..., and the COO data.frame itself was passed as the array to write, whose dim() is (nnz, 3) rather than the original matrix's (nrow, ncol). Separately, the resource dict for this layout carried no schema field, unlike every other array writer, so .readParquetArray() could never detect the 2 keycols needed to dispatch to DuckDBMatrix() on read and silently fell back to DuckDBArray() with no datacol or keycols at all. Fixed by writing the label matrix through the existing, already-schema-correct writeParquet,ANY-method (the same path used for every other array-like object) instead of a bespoke COO assembly, which fixes both problems by construction. That fix still left a gap, though: writeParquet,ANY only attaches categories (the full index->label map a reader uses to recover the true axis size) when the array already has dimnames at the point it captures them, before synthesizing seq_len() dimnames for the physical write; a spatialLabels() matrix is normally dimname-less, so no categories were recorded, and a label whose last row or column was entirely zero (dropped by the sparse COO write) came back smaller than the original. Fixed by synthesizing dimnames on the matrix (the same seq_len() values ANY would assign internally anyway) before writing it, so categories are always attached regardless of what the input happened to have. Removed the now-dead .matrixToCoordLabel(). Added round-trip regression tests: one asserting a matrix-valued label comes back as a DuckDBMatrix with the original values, and one using a label whose only nonzero entry is at row 1/col 1 to catch the trailing all-zero row/column gap (the first test alone has a nonzero in both the last row and last column, masking it). Changes in version 0.99.25 Bug fixes - writeParquet,SummarizedExperiment-method chose rowRanges(x) vs. rowData(x) for the features resource using is(x, "RangedSummarizedExperiment") alone, which is TRUE for every SingleCellExperiment regardless of whether real ranges were ever provided -- silently discarding real rowData in favor of an empty genomic_ranges_list. Fixed to check for a non-empty rowRanges(x) instead; the same check now also drives the model default. Added a regression test. Documentation - writeParquet()'s Assays-class method transposes each assay matrix and reverses indexcols/indexrefs/grid before writing (so a hive-partitioned assay is laid out sample-major, __sample__group__=.../__feature__group__=...), and prefixes each assay's on-disk directory with assay_ while keeping the Frictionless resource name as the bare assay name -- neither was documented anywhere outside an inline code comment. Added a \strong{Assays objects:} paragraph to writeParquet()'s @details covering both, found while a downstream project ported the write path to a new (non-array, streaming) source and had to reverse-engineer this from source to get partition nesting right. - Documented, on the same help page, the existing (but previously unwritten) recipe for streaming a hive-partitioned coordinate array from a source with no natural in-memory array representation: accumulate each batch into a small array slab and call writeParquet(slab, ..., append = TRUE, along = , offset = , group_offset = ) repeatedly. Also documented the one real restriction to plan around (append = TRUE requires length(grid) > 1L for the slab's own grid) and that BiocDuckDB's own writers always pass existing_data_behavior = "error" through to arrow::write_dataset (never the data-losing "delete_matching"), so the corruption case only arises for code that bypasses writeCoordArray/writeParquet entirely. - writeStreamingResource() -- an existing, exported, and already thoroughly-documented function for streaming a flat resource from a block-producing callback -- was not referenced from writeParquet()'s or writeDatapackage()'s @seealso, nor from the vignette, making it effectively undiscoverable outside its own help page. Added the cross-references, and a new "Streaming a flat resource from a cursor" vignette subsection using it in place of the vignette's previous from-memory-only example. - readParquet()'s "Supported Object Types" section now notes, per model, which resources carry a __feature__group__/__sample__group__ dimtbl column (features/samples do; sample_embeddings/feature_embeddings never do) and the assay's partition nesting order -- previously undocumented, and the closest thing to a per-model "resource contract" reference this package has. writeDatapackage()'s @param model pointed at "the storage-layout vignette" for this table, but the vignette's storage-layout section never contained one; retargeted that pointer at this section instead, since it already had the right shape. - Documented that writeCoordArray()'s default grid (used whenever writeParquet() delegates to it for an array write) budgets partitions from dim(x) alone, as if x were fully dense, and cross-referenced DuckDBArray::writeCoordArray's new "Sizing grid for large or sparse arrays" section for the density correction (setAutoBlockSize()/setAutoBlockShape()) and its hard ceiling. Added a third @examples entry and a matching vignette snippet showing those calls made before writeParquet() on a sparse matrix. Changes in version 0.99.24 Bug fixes - Found while independently verifying scoreMarkers,DuckDBMatrix-method's true.auc = TRUE path against a from-scratch oracle: three bugs in .compute_true_auc_sql_DuckDBMatrix(). First, it matched its SQL results back to R by raw row-key value against seq_len(ngenes), which only coincide when x is the full, unsubsetted matrix. On a row-subsetted DuckDBMatrix (scoreMarkers(..., subset.row = ..., true.auc = TRUE)), this either left genes as NA or silently mapped a result to the wrong output row entirely. Fixed to match against the actual row-key-to-position mapping, the same pattern already used elsewhere in this file. - The same function's threshold (lfc) shift was applied identically to every cell's value regardless of group, which is a no-op on rank order -- lfc silently had zero effect on the computed AUC for any non-zero value. Fixed to shift only the "left" group's values, matching the P(left > right + lfc) definition used everywhere else in this package's effect-size code. - The same function derived its gene universe from SELECT DISTINCT over only the stored rows of the COO table, so a gene with zero stored values across the entire matrix (a fully implicit-zero row, common in sparse single-cell data) never appeared in the output at all -- silently NA instead of the correct all-ties AUC of 0.5. Fixed by registering the full set of row keys as the gene universe instead of deriving it from the data. - .register_temp_table()'s unique-name generation (used by 7+ call sites across this file for zero-copy join temp tables) drew its "unique" suffix from sample.int(), which pulls from R's global RNG stream and only combined with second-level wall-clock resolution -- empirically, only ~43% of 20000 draws were unique. Because duckdb_register(..., overwrite = TRUE) silently overwrites rather than erroring, a collision (e.g. two calls within the same wall-clock second under a seeded/reproducible pipeline) could silently corrupt results instead of failing loudly -- the same anti-pattern independently found and fixed in the sibling DuckDBArray package (0.99.9, dbplyr::unique_table_name()). Fixed to a monotonic, PID-qualified counter, and .compute_true_auc_sql_DuckDBMatrix()'s two hand-rolled temp-table registrations (one pre-existing, one added for the gene-universe fix above) now go through this shared, fixed helper instead of duplicating the same vulnerable naming scheme inline. - correlatePairs,DuckDBMatrix-method computed its Pearson correlation (rho) but never the significance test its own documented return value ("a DataFrame of gene pairs with correlations and significance") promised -- output had no p.value/FDR at all. Added the standard Pearson correlation t-test (t = rho * sqrt((k-2)/(1-rho^2)), df = k-2) plus BH-adjusted FDR, verified to match cor.test(..., method = "pearson") exactly. Documentation also now says plainly that this is Pearson's r on x as given, not scran::correlatePairs()'s Spearman's rho on ranked expression, so results are not expected to numerically match scran's own method (the existing tests already knew and tested for this; only the package-level docs previously implied more compatibility than is true). Changes in version 0.99.23 New features - Added DuckDBIrlbaParam(), a BiocSingular::BiocSingularParam for BiocSingular::runSVD() (and so scran::fixedPCA()/scater::runPCA()/ calculatePCA()) that closes the solver-loop performance gap the initializeCpp() work below documented but did not fix. Measured directly on a real 12,500-cell, 200-HVG benchmark: the ordinary BSPARAM = IrlbaParam() path took ~16.2s on a DuckDBMatrix versus ~0.3s in-memory (~51x), because irlba's Lanczos loop calls the SQL-pushdown %*% once per solver iteration, paying real query-construction cost every time. DuckDBIrlbaParam() instead materializes the matrix once (reusing loadIntoMemory()'s size-gated .duckdb_seed_to_sparse_matrix()) and calls BiocSingular::runIrlbaSVD() directly on the materialized matrix, verified to reproduce identical singular values to the lazy path at roughly 300x the speed in this session's testing (0.084s vs 26.2s on a synthetic 3000x500 case). Confirmed on the same real 12,500-cell, 200-HVG benchmark that motivated this fix: calculatePCA on DuckDBMatrix dropped from ~16.2s to ~0.58s, beating HDF5Array's ~4.45s by about 7.6x and landing within 1.8x of the ~0.33s in-memory dgCMatrix baseline. Falls back to ordinary IrlbaParam behavior whenever the fast path doesn't apply (not a DuckDBMatrix, non-zero-filled seed, or over "memory_limit"), including reproducing that path's own pre-existing failure modes (e.g. a non-zero-filled seed) identically rather than papering over them. - Registered a beachmat::initializeCpp() method for DuckDBArraySeed (initializeCpp.R, initializeOptions.R, loadIntoMemory.R), giving compiled C++ code that goes through beachmat/tatami (e.g. BiocSingular's compute_center/compute_scale) a fast native path instead of always falling back to beachmat's generic "unknown matrix" block-processing path. Unlike beachmat.hdf5/beachmat.tiledb, there is no tatami_duckdb C++ library for genuine on-demand, out-of-core reads, so this materializes the seed's COO table into an in-memory sparse matrix via a single bulk SQL pull (size-gated by the new "memory_limit" option, default 2 GB) and delegates to beachmat's own native initializeCpp,dgCMatrix-method. Whenever the fast path doesn't apply (a non-zero-filled seed, more than 2 keyed dimensions, or an estimated size over the limit), it falls back to beachmat's existing generic path rather than erroring, so it can never turn previously-working (if slower) code into a failure. This closes part of the solver-loop performance gap noted below: initializeCpp itself is unaffected by it (it never speeds up irlba's Lanczos loop, which calls %*% directly), but it does speed up anything that already calls initializeCpp. - Deliberately does not cache the materialized pointer across calls with beachmat::checkMemoryCache, unlike beachmat.hdf5/beachmat.tiledb. Verified empirically that caching here is unsafe: two DuckDBMatrix objects built from the same underlying query (the common case, e.g. re-reading the same release twice) render identical SQL and hit the same cache entry, but reusing that cached pointer across a later, logically separate computation silently corrupted results (reproduced directly: running scran::fixedPCA() once, then scater::runPCA() on a freshly-built DuckDBMatrix over the same data, diverged from the dense oracle only when the second call hit the first call's cached pointer, and matched exactly with caching removed). Every call re-materializes instead; this costs a fraction of a second for a memory_limit-sized matrix. Testing - Confirmed and locked in with a regression test (test-DuckDBMatrix-svd.R) that BiocSingular::runSVD(), scran::fixedPCA(), and scater::runPCA() already compute a numerically correct PCA on a DuckDBMatrix, including on a genuinely sparse (COO-encoded, majority implicit-zero) fixture, matching a dense irlba oracle to ~1e-8/1e-14 depending on the comparison. - Documented a real, unaddressed performance caveat found during this investigation: the per-%*%-call query-construction overhead is negligible in isolation (~0.04s/call) but adds up across the ~hundreds of calls an iterative Lanczos solver makes, k=10 on a 3000x500 sparse matrix took ~22s versus ~0.07s to compute center/scale. This is a solver-loop scalability gap worth optimizing for genomics-scale matrices, not a correctness issue. Changes in version 0.99.22 Documentation - The benchmarking vignette's "your build's bundled results" phrasing wrongly implied the shipped benchmark table could vary per installation. Verified the actual mechanics: load_vignette_timings() reads inst/scripts/benchmark_results.rds from the installed package, while run_scran_scuttle_benchmarks.R writes its output to the working directory, not that location, so running the script does not, by itself, change the table; every installation of a given release bundles the same fixed, precomputed results file. Removed the "your build" framing from the Summary section and moved the (now corrected) explanation of how a maintainer would actually refresh the bundled numbers down to "Running your own benchmarks," where the same inaccuracy ("which this vignette renders") was also present and is now fixed. Changes in version 0.99.21 Bug fixes - c(object.size(x), object.size(y)) strips the object_size class, so the introduction vignette's format(..., units = "MB") call silently ignored units and printed raw byte counts instead. Fixed by formatting each object.size() result separately (preserving the class) before combining them into the displayed vector. - spatialElementJoin()'s documentation claimed it joins "arbitrary element pairs," but the underlying DuckDBSpatial::layerSpatialMatch() primitive only ever supports a points element against a shapes/geometry element, never points-vs-points or shapes-vs-shapes. Calling it with an unsupported combination surfaced as a cryptic low-level DuckDB Binder Error rather than a clear message. - system("nproc", intern = TRUE, ignore.stderr = TRUE) in inst/scripts/run_scran_scuttle_benchmarks.R replaced with system2("nproc", stdout = TRUE, stderr = FALSE), per Bioconductor's guidance to prefer system2(). Documentation - The introduction vignette's airway example now guards on requireNamespace("airway", quietly = TRUE) so the vignette still compiles when the Suggested airway package is not installed. - Improved both metadata-skip warning messages in .serializeMetadataValue() (writeParquet.R) to name the metadata item's class and give a plain-language reason instead of leaking the raw internal S4 dispatch error text. Writing the flagship airway example (writeParquet(airway, se_path)) emits one of these warnings, since airway's MIAME metadata object is not representable as JSON, a data frame, or an array-like object; the warning is expected and the metadata item is correctly skipped, but the old message was confusing. - Rewrote the "Targeting the storage contract" section (renamed "Advanced: writing and attaching resources directly") in plain language for a general reader: defined "Frictionless Data Package" concretely instead of assuming familiarity, dropped "base-conformant"/"ingest/attach" jargon, and added a note that first-time readers can skip the section. - Reworded the "Results" section's lead sentence to no longer read like a code comment; the ambiguous "consult the rendered table" sentence is clarified; "Main takeaways" is now a "Summary" prose section that explains why each result happens instead of restating the table, and answers a reviewer question about performance tips; and the ambiguous "generics run unchanged on disk" phrasing is rewritten rather than repunctuated. - Added a real, runnable @examples block for spatialElementJoin(), replacing the commented-out pseudocode. Changes in version 0.99.20 Bug fixes - .compute_true_auc_sql_DuckDBMatrix() (the scoreMarkers(..., true.auc = TRUE) path) interpolated group labels, arbitrary user-supplied text from the groups factor's levels, straight into raw SQL (WHERE group_label IN ('...', '...')) with no escaping. A label containing a single quote (not an exotic case for a cell-type or condition name) produced malformed SQL. Found during an independent code-quality review of the package's for loops; every other raw-SQL-building spot in this file only interpolates internal identifiers or integers, never free-text user data, so this was the one place that departed from that safer pattern. Fixed by quoting both group labels with DBI::dbQuoteString() before interpolation. Verified with a group label containing an apostrophe. Documentation - Fixed `r CRANpkg("airway")` to `r Biocpkg("airway")` in the introduction vignette; airway is a Bioconductor experiment-data package, not a CRAN package, so the old macro rendered a broken link. - Fixed two comma splices in the introduction and benchmarking vignettes. - Added a one-sentence explanation of DuckDBMatrix()'s datacol/keycols arguments where the constructor first appears in the introduction vignette, since neither is explained anywhere else in the vignette. Changes - Vectorized the Monte-Carlo simulation loop in .generate_poisson_values_DuckDBMatrix() (used by modelGeneVarByPoisson's trend fitting): one batched rpois()/rnbinom() draw across all simulation points instead of up to npts (default 1000) separate calls. - Removed a dead accumulator variable (gene_means_by_block) in .compute_blocked_stats_DuckDBMatrix() that was written but never read. - Collapsed 4 per-group/per-column for loops in numDetectedAcrossFeatures(), sumCountsAcrossFeatures(), and summarizeAssayByGroup() (sum and detected branches) in DuckDBMatrix-scuttle.R into one-line matrix broadcasts, matching the t(t(...)) idiom already used elsewhere in the same file. Changes in version 0.99.19 Bug fixes - The categorical value-label decoder in .readParquetResource() (used to reconstruct dictionary-encoded key columns, e.g. sample_map's assay column) is built with sapply() specifically because its output must retain the names FUN returns (value -> label). During an independent code-quality review, this call was one of several converted to vapply() to address BiocCheck's sapply() NOTE; vapply() never takes names from what FUN returns when FUN.VALUE is an unnamed scalar template, so the conversion silently produced an unnamed vector. This corrupted keycol decoding, which surfaced as sampleMap reconstruction failures ("not all ExperimentList samples are found in the sampleMap") on writeParquet()/readParquet() round-trips of spatial MultiAssaySpatialExperiment objects. Caught by the full test suite before being committed; reverted to sapply() with a comment explaining why, and left as the one deliberate exception BiocCheck still flags. Documentation - Fixed roughly 69 \link{} cross-references across 16 man pages (readParquet, writeParquet, DuckDBDualSubset, the DuckDBMatrix/DuckDBDataFrame/SingleCellExperiment method families, and the MultiAssaySpatialExperiment query helpers) that pointed at classes or functions defined in other packages (DuckDBArray, DuckDBDataFrame, DuckDBGRanges, MultiAssaySpatialExperiment, S4Vectors) without a package anchor, which R CMD check --as-cran flags as a NOTE. Changes - Added S4Arrays to Suggests:; it was already an undeclared dependency of two test files (test-writeParquet-coord-append.R, test-writeParquet-roundtrip-append.R). - Removed 6 dead internal functions with no remaining call sites: .extractIntType, .fieldToDuckDBCast (fieldtypes.R); .filterSpatialLayerByInstances, .filterSpatialLayersByInstances, .spatialExtent (MultiAssaySpatialExperiments-internals.R); .prefixSeq (writeParquet.R). - Collapsed 4 paste()/paste0()-built strings passed to stop()/warning() into single string literals (DuckDBMatrix-scran.R, writeStreamingResource.R), matching BiocCheck's preference for a single literal over a dynamically assembled condition message. - Converted 8 of 9 flagged sapply() calls in readParquet.R to vapply() for a guaranteed return type and length; the 9th is documented above. Changes in version 0.99.18 Bug fixes - writeParquet.GenomicRanges() silently wrote positional numbers ("1", "2", ...) instead of a named GenomicRanges' real feature names as the __name__ display column, whenever the installed GenomicRanges was 1.65.2 or later. GenomicRanges 1.65.2 rewrote as.data.frame.GenomicRanges(): it used to default row.names to names(x) when not supplied; the new version hardcodes row.names = NULL internally (an explicitly-passed row.names argument is silently discarded too) and puts names(x) in a new names column instead. writeParquet.GenomicRanges()'s df <- as.data.frame(x, optional = TRUE) relied on the old convention to carry real names into df's row names, which then became the written __name__ column; with the new convention, df's row names were the default numeric sequence instead. Reading a RangedSummarizedExperiment written this way surfaced as rowRanges() names coming back wrong, and then, one step further, name-based row indexing (x[names(x)]) failing outright with a DuckDB Conversion Error trying to match the real names against an integer keycol. Root-caused by installing the exact upstream GenomicRanges commit and reproducing the original failure byte-for-byte; fixed by explicitly setting rownames(df) <- names(x) (and dropping the new names column) after the conversion, which works against both old and new GenomicRanges. writeParquet.GenomicRangesList() was checked and already set rownames() explicitly, so it was unaffected. - checkDuckDBGRanges() (this package's own test helper, duplicated from DuckDBGRanges) had the same reliance on the old as.data.frame.GenomicRanges() convention in its own reference comparison; fixed the same way. See DuckDBGRanges 0.99.7's NEWS.md for the sibling fix to its copy of this helper. Changes in version 0.99.17 Bug fixes - .readParquetMAE() silently misbehaved on two edge cases in a multi_assay_experiment package: a missing subjects resource collapsed file.path(path, NULL) to character(0), reaching DuckDBDataFrame() with an opaque downstream error instead of a clear one; a missing sample_map resource indexed into a NULL resource the same way. subjects now errors immediately with a message naming the missing resource. A missing sample_map is now handled by deriving a real map, the same way MultiAssayExperiment()'s own constructor does when its sampleMap argument is omitted (matching column names against colData row names) rather than falling back to an empty DataFrame, since an empty sampleMap only satisfies MultiAssayExperiment's own .checkSampleNames() validity check when every experiment is also empty. That derivation now lives in .newMAE() (this package's own internal constructor, called from exactly one place) rather than in .readParquetMAE() itself, so any future caller passing sampleMap = NULL gets the same behavior for free, without .newMAE() taking on MultiAssayExperiment()'s own .harmonize()/full validObject() cost, which this package's writer output does not need paying again on read. Changes in version 0.99.16 Bug fixes - writeStreamingResource() created its _INCOMPLETE marker after part 0 had been written, so a crash during part 0, the very case the marker exists to catch. It left no marker at all, contradicting the documented behavior. It is now created immediately before part 0, along with the directory to hold it. Changes in version 0.99.15 - Reading a MultiAssaySpatialExperiment loaded every raster-backed label twice: correctly into spatialLabels(), and again into spatialImages() under the unstripped name sample_labels_. Only an array-valued label takes the spatial_label_coord branch on write; a raster-backed one is written with the same spatial_raster_ref layout an image uses, so selecting the image resources by layout alone could not tell the two apart. The label branch already narrowed its own selection by name; the image branch now does the same. - readParquet() could not read a package declaring model = "experiment_list"; it failed with subscript out of bounds. The dispatch passed the whole datapackage.json manifest to .readParquetExps(), which takes the resource list rather than the manifest (its two other callers, the altExps path and .readParquetMAE(), both pass a filtered resource list). It therefore iterated the manifest's top-level entries and tried to read a name field off $schema. It is now handed package[["resources"]]. experiment_list is one of the reader values named in the profile's model description, and writeParquet() has had an ExperimentList method throughout. The gap went unnoticed because the round-trip suite explicitly skipped ExperimentList (a note to that effect sat in test-parquet-roundtrip.R); it is now covered. Changes in version 0.99.14 Testing - Loading the airway Suggests package in tests/testthat/setup.R is now conditional, and the airway-dependent test files (test-DuckDBMatrix-scuttle.R, test-DuckDBMatrix-scran.R) and test block (test-parquet-roundtrip.R) call skip_if_not_installed("airway"). Previously the unconditional data(airway, ...) call in setup.R would fail and abort the entire test suite (not just the airway-dependent tests) when airway was not installed. Same class of issue as found in the DuckDBArray Bioconductor review. - test-MultiAssaySpatialExperiment.R calls skip_if_not_installed("sf"), the Suggests package its fixtures build geometries with. Changes in version 0.99.13 Follow-up review changes - Updated the sole caller of the shared internal generic (a round-trip test) to the undotted name after the Bioconductor-review rename in DuckDBDataFrame (.has_row_number -> has_row_number; guideline: functions starting with . should not be exported). Requires the renamed version of DuckDBDataFrame. Documentation - Replaced em dashes with commas or colons in the vignettes. Changes in version 0.99.12 Documentation - Added URL and BugReports fields to DESCRIPTION. - Added a package-level man page (?BiocDuckDB). - Removed the redundant library(BiocStyle) call from the vignettes (the BiocStyle::html_document output and ::-qualified helpers apply the style without loading the package). Applied suite-wide for consistency with the Bioconductor review of DuckDBDataFrame. Changes in version 0.99.11 Enhancements - readParquet() now assembles multi_assay_experiment and multi_assay_spatial_experiment objects directly with S4Vectors::new2(check = FALSE) instead of calling the MultiAssayExperiment() / MultiAssaySpatialExperiment() constructors. Those constructors re-derive sample-name set equality at construction time via a locale-collation sort over the full, cell-grain sampleMap, which dominates read time on large products (tens of millions of sampleMap rows -- e.g. a cell-resolved spatial atlas took ~22 min to read, over 60% of it in that redundant sort). The datapackage already carries writeParquet()'s write-time integrity guarantee (a DuckDB anti-join proving every sampleMap primary is a subjects rowname), so the constructor-time re-check is redundant; the assembled object still passes validObject(). The direct assembly also skips harmonization, so unreferenced colData rows are retained rather than dropped on read (write with subset_subjects_to_referenced upstream to avoid them). Testing - The multi_assay_experiment and multi_assay_spatial_experiment round-trip tests now assert validObject() on every object returned by readParquet(). Because the read path assembles with new2(check = FALSE) (no constructor-time validity), these explicit checks guard against a readParquet()-assembled object being structurally invalid. Changes in version 0.99.10 Enhancements - writeParquet() now serializes a MultiAssaySpatialExperiment's spatialMap as a monomorphic bridge over a conformed element-instance registry (spatial_element_registry), so the observation-to-spatial-element association becomes real, single-target foreign keys instead of an opaque polymorphic reference. A new spatial_element_registry resource enumerates every points/shapes instance across all layers with an integer __element__ spine; each typed spatial layer foreign-keys that spine; and spatial_map declares two foreign keys -- the element side (__element__ to spatial_element_registry) and the observation side ((assay, colname) to sample_map, which encodes the spatialMap validity rule as a real composite FK). The __element__ spine is a serialization-internal column: readParquet() reconstructs the object-model spatialMap and the spatial layers without it, so the round-trip is unchanged. Bug fixes - writeParquet() emitted a coord_array index column's category label as a one-element array rather than a string, so a datapackage carrying an assay with dimnames failed validation against the bundled Frictionless profile. The label is now a string, and such packages conform. Changes in version 0.99.9 Enhancements - writeParquet() now declares dimension foreign keys for neighbor graphs and reducedDim embeddings, so a datapackage.json losslessly serializes the object model's relational structure rather than only the assay's. A colPairs / rowPairs graph_edges resource declares both edge endpoints (from / to) as foreign keys into the sample / feature dimension's crossing key (a role-playing self-reference), and a reducedDims / rowLoadings embedding_table resource declares its key column as a foreign key into its dimension table (an outrigger). The additions are additive and reader-neutral: the reader reconstructs graphs and embeddings from graphEdges / the Parquet schema as before, and the emitted package still validates against the bundled Frictionless profile. Changes in version 0.99.8 New features - writeStreamingResource() now guards a multi-part write with an in-progress marker: an _INCOMPLETE file is dropped into the resource directory once part 0 creates it and removed once the stream finishes. A completed resource is therefore a clean directory of only Parquet parts that reads normally, while an interrupted write leaves the marker behind; because the marker is not a Parquet file, reading the incomplete directory fails loudly instead of silently returning the partial parts. This makes "readable only when complete" hold by construction with no change to the reader. Enhancements - The flat data.frame Parquet write path now pins the row-group size (arrow::write_parquet(chunk_size = ...), default 491520 rows, matching the coordinate-array writer) instead of relying on the Arrow default. A large or streamed flat resource is written with bounded row groups, so range predicates prune at row-group granularity and reads parallelize across groups; a caller may still override chunk_size. Changes in version 0.99.7 New features - writeStreamingResource() streams a larger-than-memory table to a single flat, multi-part Parquet resource by pulling it block-by-block from a producer callback (blocks(i) returning the i-th data.frame or NULL when exhausted). It owns the streaming bookkeeping that large producers otherwise hand-roll on top of writeParquet(): the running row offset, the part index and zero-padded part_digits, the append flag, the positional dimtbl slice per block, and index_max threading so the __index__ column stays one consistent (possibly > 2^31) integer type across parts. It also performs the post-write integrity checks — a fatal partition-alignment guard when the streamed row count disagrees with the dimension table, a coverage warning against expected_rows, and a narrowing-floor warning when a small part 0 is followed by more parts without index_max. The returned descriptor (with n_rows) is ready for writeDatapackage(). Changes in version 0.99.6 Bug fixes - Reading a graph_edges resource no longer materializes the __index__ keycol. .readParquetGraphEdges() now constructs the graph with a row_number key (keycol = NULL) instead of the schema's __index__ keycol. A level-discovered keycol pulls every distinct edge id into R and sorts it, which overflowed bit64's radixsort ("long vectors not supported") for a graph with more than ~2.1e9 edges. A graph edge is identified by its from/to endpoints, not its row position, so the row-number key reconstructs the same graph without the scan. Changes in version 0.99.5 - .writeDataFrameParquet() now fails loudly when a part's __index__ would exceed the 32-bit range but index_max was not supplied, instead of silently writing a floating-point (float64) key spine. Producers streaming past 2^31 rows must declare index_max (or Inf) so the column is typed int64 consistently across parts. - writeDatapackage() validates its inputs at the write seam (every resource must be a descriptor with a single-string name and path; names must be unique), so a malformed manifest fails fast at write time rather than surfacing as an obscure error at read time. Changes in version 0.99.4 Bug fixes - writeParquet() (flat / data-frame path) can now write a resource with more than ~2.1e9 rows without the __index__ column overflowing 32-bit integers. A new optional index_max argument declares the resource's total index range; when it exceeds the 32-bit limit the __index__ column is typed as a 64-bit integer up front (part 0 included) so every streamed append part shares one type, instead of narrowing part 0 to int32 and overflowing later parts on cast (mirrors the coord-array max_dim typing). Pass index_max = Inf when the total row count is unknown before streaming (e.g. graph edges). Small resources are unaffected --- the index still narrows to the smallest integer type. Requires the companion DuckDBDataFrame validateAppendOffset() fix. Changes in version 0.99.3 New features - Added the exported writeDatapackage() function, which assembles and writes a Frictionless datapackage.json envelope from a list of resource descriptors. The experiment-level writeParquet() methods (SummarizedExperiment, MultiAssayExperiment) now single-source their manifest assembly through it, and producers that build resources incrementally --- streaming a dataset too large to hold in memory, or promoting from another store --- can emit a conformant manifest without reconstructing an in-memory Bioconductor object. NULL descriptors (returned by append/streaming parts) are dropped, so accumulated resource lists can be passed straight through. This is the write half of the ingest contract; the read half is readParquet() together with the DuckDBMatrix()/DuckDBArray()/DuckDBTable() constructors, which attach existing Parquet in place. Documentation - Documented the storage layout as a targetable Frictionless contract in the package vignette (the new "Targeting the storage contract" section), covering writeDatapackage() for assembling a manifest and the DuckDB-backed constructors for attaching existing coord-array Parquet in place. Changes in version 0.99.2 Documentation - Added \value sections (roxygen @return) to the DuckDBDataFrame-spatial, DuckDBDualSubset-class, DuckDBMatrix-scran, DuckDBMatrix-scuttle, and MultiAssaySpatialExperiment-spatial man pages, documenting the values returned by the spatial query, dual-subset, scran, scuttle, and MASE spatial I/O methods. Resolves the R CMD BiocCheck "missing \value" WARNING. Changes in version 0.99.1 Bug fixes - The spatial-map query functions (linkSpatialMap(), spatialViews(), spatialCoordinateSystems(), validateSpatialMap()) now have runnable examples backed by a minimal spatial MultiAssaySpatialExperiment fixture bundled under inst/extdata/spatial_mase/ (generated by inst/scripts/make-spatial-mase-fixture.R). This raises the share of exported-object man pages with runnable examples above the 80% BiocCheck threshold, resolving the R CMD BiocCheck ERROR. Changes in version 0.9.29 Bug fixes - writeParquet() now right-sizes the internal integer index columns of data-frame resources (the __index__/__sample__/__feature__ key spine and graph-edge from/to) to their smallest Arrow integer type. This narrowing was previously unreachable: it was gated on !flat_part, but setupFlatParquetWrite() always resolves flat_part = TRUE, so data-frame index columns were written as int32. The coord-array assay path already narrowed, so only the tabular resources were affected. - Scoped to internal index columns to match scibis (_optimize_integer_columns / _narrow_index): user data columns and Hive partition-group columns keep their declared dtype, so a write→read round-trip is dtype-preserving. - On append, each part's index columns are pinned to the existing part-0 schema (via reconcileParquetSchema(), whose result was previously discarded) so every part of a streamed resource shares one integer width. - The recorded arrowType in each resource's schema now reflects the narrowed width, matching the physical Parquet. Changes in version 0.9.28 New features - readParquet() can now read a dataset from remote object storage (s3://, gs://, http(s)://, …). The datapackage.json sidecar is fetched over DuckDB httpfs (jsonlite::read_json cannot fetch object-storage schemes) after DuckDBDataFrame::configureCloud() installs/loads httpfs and applies any s3_* credentials; the resource reads then prune row groups over the VFS the same as on local disk. Local paths are unchanged. Changes - writeParquet() refuses a remote object-storage path with an actionable error ("write to a local directory and upload it"). Object stores have no atomic directory swap and the append/refuse-to-overwrite guards rely on local file existence, so BiocDuckDB writes stay local + whole-object; upload the written directory afterwards (e.g. aws s3 cp --recursive). Changes in version 0.9.27 Changes - The test harness pins DuckDB to a single thread (options(DuckDBDataFrame.threads = 1L) in setup.R) so parallel floating-point reductions accumulate in a fixed order and tight-tolerance expectations stay reproducible run-to-run. This is a test-only change; real sessions use all cores. Changes in version 0.9.26 New features - Cross-element DuckDB query layer over a MultiAssaySpatialExperiment whose spatial layers are DuckDB-backed (R/MultiAssaySpatialExperiment-query.R). MASE stays DuckDB-free; this is BiocDuckDB's DuckDB engine operating on a MASE, using DuckDBSpatial (Suggests) for spatial SQL. - spatialViews() registers each spatial layer and the spatialMap junction as on-the-fly DuckDB temp views (lazy layers as views over their rendered SQL; no materialization). - linkSpatialMap() links assay observations to their spatial layer rows through the full spatialMap key (assay, colname, element_type, region, instance_id), returning a lazy DuckDBDataFrame. - validateSpatialMap() checks referential integrity via DuckDB anti-joins (orphan instance_id/colname, unknown layer, duplicate instance_id); reports by default, errors with strict = TRUE. - spatialElementJoin() runs an ST_* cross-element spatial join (via DuckDBSpatial), first aligning both elements into a common coordinate system through the coordinate-transform graph when coordinate_system is given. - spatialCoordinateSystems() and per-element transforms stored in metadata(mase)$transforms ("/" -> {cs -> transform}, RFC-5-shaped) round-trip through the standard MASE metadata annotations. - writeParquet(..., cluster_by = ) threads a clustering key (DuckDBDataFrame::zorder() / hilbert(), or a character vector) through every writeParquet method to the primitive writers, so rows are physically ordered on write for DuckDB row-group zonemap pruning. The lazy DuckDBTable/DuckDBDataFrame path lowers it SQL-side (no materialization); the materializing data.frame/DataFrame path reorders in memory via DuckDBDataFrame::clusterSort(). Requires DuckDBDataFrame (>= 0.9.27). Bug fixes - Writing an in-memory geometry table (an sf / DataFrame with a geometry column, e.g. a ShapesLayerList layer) to Parquet no longer errors with "flat append ('append', 'part') is not supported for sf objects". Since flat writes became the default, every such write was a flat part 0, which the sf branch of .writeDataFrameParquet mistook for an append/part continuation. The guard now rejects only a genuine append or a subsequent part (part > 0), matching the lazy geometry write path, so a normal geometry write goes through DuckDBSpatial::writeGeoParquet. Changes in version 0.9.23 Bug fixes - readParquet() now restores factor columns (including ordered factors) in the flat-table read paths, using the categories/categoriesOrdered recorded in the product schema on write. Previously these columns came back as character. (Factor restoration inside GRanges/SelfHits mcols is not yet covered.) Documentation - Restructured the vignettes into a user-first set, replacing Constructing Experiment Objects with BiocDuckDB and scuttle and scran integration for DuckDBMatrix: - Introduction to BiocDuckDB --- the writeParquet()/readParquet() workflow, the Frictionless/coordinate storage layout, operations pushed to SQL, genomic coordinates, single-cell data, and the filter-realize-analyze pattern. - Benchmarking BiocDuckDB --- the SQL-optimized scuttle/scran methods on a DuckDBMatrix compared with in-memory and HDF5Array, rendered from precomputed full-scale results (10x 1.3M brain cells) so the vignette builds quickly. - Added inst/scripts/ with the offline benchmark generator (run_scran_scuttle_benchmarks.R) and the vignette table helpers (make_timings_table.R), following the HDF5Array performance-vignette precompute pattern. - Rewrote the README.