Validation and Sharing Indexes

Persisted indexes are most useful when they can be reopened safely later or shared with collaborators without guessing how they were created.

bigANNOY v3 addresses that problem with two ideas:

• each Annoy index has a sidecar metadata file
• persisted indexes can be checked with annoy_validate_index() before use

This vignette focuses on those operational safeguards.

Load the Packages

library(bigANNOY)
library(bigmemory)

Create a Small Persisted Example

We will build a small Euclidean Annoy index and keep all of its files inside a temporary working directory.

share_dir <- tempfile("bigannoy-share-")
dir.create(share_dir, recursive = TRUE, showWarnings = FALSE)

ref_dense <- matrix(
  c(
    0.0, 0.0,
    1.0, 0.0,
    0.0, 1.0,
    1.0, 1.0
  ),
  ncol = 2,
  byrow = TRUE
)

ref_big <- as.big.matrix(ref_dense)
index_path <- file.path(share_dir, "ref.ann")

index <- annoy_build_bigmatrix(
  ref_big,
  path = index_path,
  n_trees = 20L,
  metric = "euclidean",
  seed = 77L,
  load_mode = "lazy"
)

index

#> <bigannoy_index>
#>   path: /var/folders/h9/npmqbtmx4wlblg4wks47yj5c0000gn/T//RtmpBEyDSE/bigannoy-share-b1f63b58f54f/ref.ann
#>   metadata: /var/folders/h9/npmqbtmx4wlblg4wks47yj5c0000gn/T//RtmpBEyDSE/bigannoy-share-b1f63b58f54f/ref.ann.meta
#>   index_id: annoy-20260327203934-c5f081efa033
#>   metric: euclidean
#>   trees: 20
#>   items: 4
#>   dimension: 2
#>   build_seed: 77
#>   build_threads: -1
#>   build_backend: cpp
#>   load_mode: lazy
#>   loaded: FALSE
#>   file_size: 1056
#>   file_md5: c5f081efa03365a2af6692ec9950b0f9
#>   prefault: FALSE

At this point the key persisted assets are:

• the Annoy index file at index$path
• the sidecar metadata file at index$metadata_path

What the Metadata Records

The metadata file is a small DCF document that records enough information to make later reopen and validation steps safer.

metadata <- read.dcf(index$metadata_path)
metadata[, c(
  "metadata_version",
  "package_version",
  "annoy_version",
  "index_id",
  "metric",
  "n_dim",
  "n_ref",
  "n_trees",
  "build_seed",
  "build_threads",
  "build_backend",
  "file_size",
  "file_mtime",
  "file_md5",
  "load_mode",
  "index_file"
)]

metadata_version	package_version	annoy_version	index_id	metric	n_dim	n_ref	n_trees	build_seed	build_threads	build_backend	file_size	file_mtime	file_md5	load_mode	index_file
3	0.3.0	0	annoy-20260327203934-c5f081efa033	euclidean	2	4	20	77	-1	cpp	1056	2026-03-27T20:39:34Z	c5f081efa03365a2af6692ec9950b0f9	lazy	ref.ann
3	0.3.0	0	annoy-20260327203934-c5f081efa033	euclidean	2	4	20	77	-1	cpp	1056	2026-03-27T20:39:34Z	c5f081efa03365a2af6692ec9950b0f9	lazy	ref.ann
3	0.3.0	23	annoy-20260327203934-c5f081efa033	euclidean	2	4	20	77	-1	cpp	1056	2026-03-27T20:39:34Z	c5f081efa03365a2af6692ec9950b0f9	lazy	ref.ann

The most important fields operationally are:

• metric, n_dim, and n_ref, which describe what the index represents
• file_size, file_mtime, and file_md5, which summarize the current Annoy file
• index_file, which records the expected basename of the .ann file
• index_id, which gives the persisted artifact a stable identifier

Validate Before You Use a Persisted Index

The safest default is to validate a reopened or long-lived index before using it for important downstream work.

validation <- annoy_validate_index(
  index,
  strict = TRUE,
  load = TRUE
)

validation$valid

#> [1] TRUE

validation$checks[, c("check", "passed", "severity")]

check	passed	severity
index_file	TRUE	error
metric	TRUE	error
dimensions	TRUE	error
items	TRUE	error
file_size	TRUE	error
file_md5	TRUE	error
file_mtime	TRUE	warning
load	TRUE	error

With strict = TRUE, any failed error-severity check stops immediately. With load = TRUE, validation also confirms that the index can actually be opened successfully.

What Counts as an Error Versus a Warning

Not every check has the same severity:

• checksum and file-size mismatches are treated as errors
• metric, dimension, and item-count mismatches are treated as errors
• file modification time is currently treated as a warning

That distinction is visible in the validation report.

Reopen the Index as a Separate Session Object

In a later R session, you would normally reattach the persisted index with annoy_open_index() or annoy_load_bigmatrix().

reopened <- annoy_open_index(
  path = index$path,
  load_mode = "lazy"
)

annoy_is_loaded(reopened)

#> [1] FALSE

annoy_validate_index(reopened, strict = TRUE, load = TRUE)$valid

#> [1] TRUE

annoy_is_loaded(reopened)

#> [1] TRUE

This gives you a clean session-level controller around the same persisted files. The reopened object can now be searched, validated again, or explicitly closed.

Sharing Checklist

When sharing an index with another user, machine, or later analysis step, keep the following artifacts together:

• the .ann file
• the .meta sidecar file
• any bigmemory descriptor files needed to reconstruct the reference or query workflow around the index

In practice, it is best to think of the .ann and .meta files as one unit.

Simulate Sharing by Copying the Persisted Files

To mimic transferring an index to another location, we will copy both files into a separate directory and reopen the copy.

shared_dir <- tempfile("bigannoy-shared-copy-")
dir.create(shared_dir, recursive = TRUE, showWarnings = FALSE)

shared_index_path <- file.path(shared_dir, basename(index$path))
shared_metadata_path <- file.path(shared_dir, basename(index$metadata_path))

file.copy(index$path, shared_index_path, overwrite = TRUE)

#> [1] TRUE

file.copy(index$metadata_path, shared_metadata_path, overwrite = TRUE)

#> [1] TRUE

shared <- annoy_open_index(
  path = shared_index_path,
  load_mode = "lazy"
)

shared_report <- annoy_validate_index(
  shared,
  strict = TRUE,
  load = TRUE
)

shared_report$valid

#> [1] TRUE

This is the basic “ship the index and reopen it elsewhere” workflow.

Non-Strict Validation for Diagnostics

Sometimes you do not want an immediate error. You want a report first so you can inspect what failed and decide whether to stop, rebuild, or repair the metadata.

To demonstrate that path, we will deliberately corrupt the copied metadata by replacing the recorded checksum with a wrong value.

bad_metadata <- read.dcf(shared_metadata_path)
bad_metadata[1L, "file_md5"] <- "corrupted"
write.dcf(as.data.frame(bad_metadata, stringsAsFactors = FALSE), file = shared_metadata_path)

shared_bad <- annoy_open_index(shared_index_path, load_mode = "lazy")
bad_report <- annoy_validate_index(
  shared_bad,
  strict = FALSE,
  load = FALSE
)

bad_report$valid

#> [1] FALSE

bad_report$checks[, c("check", "passed", "severity")]

check	passed	severity
index_file	TRUE	error
metric	TRUE	error
dimensions	TRUE	error
items	TRUE	error
file_size	TRUE	error
file_md5	FALSE	error
file_mtime	TRUE	warning

This pattern is especially helpful in higher-level tools that want to show a validation report instead of terminating immediately.

Strict Validation as a Gate

For production-style workflows, strict = TRUE is usually the better default because it turns a failed validation into an immediate hard stop.

strict_error <- tryCatch(
  {
    annoy_validate_index(shared_bad, strict = TRUE, load = FALSE)
    NULL
  },
  error = function(e) conditionMessage(e)
)

strict_error

#> [1] "Recorded file checksum matches the current Annoy file checksum."

The exact message may vary depending on which error-severity check fails first, but the key point is that the corrupted metadata is no longer silently accepted.

A Common Sharing Pitfall: Renaming Only the .ann File

The metadata records the expected basename of the Annoy file in index_file. That means you should generally keep the .ann file and the .meta file paired and consistent.

If you rename the .ann file without updating or regenerating the metadata, annoy_open_index() will reject the mismatch.

renamed_path <- file.path(shared_dir, "renamed.ann")
file.copy(shared_index_path, renamed_path, overwrite = TRUE)

#> [1] TRUE

rename_error <- tryCatch(
  {
    annoy_open_index(renamed_path, metadata_path = shared_metadata_path)
    NULL
  },
  error = function(e) conditionMessage(e)
)

rename_error

#> [1] "Annoy metadata does not match the supplied index file path"

That guard is useful because it prevents accidentally pairing the wrong Annoy file with the wrong metadata file.

Recommended Sharing Pattern

For practical collaboration, a good pattern is:

1. build the index with annoy_build_bigmatrix()
2. keep the generated .ann file and .meta file together
3. move or copy them as a pair
4. reopen with annoy_open_index() or annoy_load_bigmatrix()
5. run annoy_validate_index() before important analysis
6. only trust the index for downstream search once validation passes

If your larger workflow depends on file-backed bigmemory data, keep the descriptor files alongside the matrices they describe as well.

Recap

bigANNOY v3 makes persisted indexes safer to reuse and share by giving them:

• a sidecar metadata file
• a stable index identifier
• recorded file signatures and build settings
• explicit validation with strict and non-strict modes

The practical takeaway is simple: treat the .ann file and the .meta file as a pair, reopen them intentionally, and validate before you trust them.