Skip to contents

Build an experimental file-backed full KNN graph

Source: R/massive_knn.R
massive_full_knn_graph.Rd

Exact search scans all reference blocks for each query block. method = "hnsw_sharded" instead builds one bounded native HNSW index per reference shard, searches all query blocks against it, and merges candidates into the file-backed output. It is approximate; its target recall is not measured unless audit_rows is positive. Multiple shard passes can be I/O-heavy. CUDA method = "exact" builds and queries bounded cuVS reference shards. It also scans the full query source per shard, so this exact route is a correctness baseline rather than a billion-row ANN solution. A CUDA method = "ivf_sharded" uses the existing native cuVS IVF-Flat index within each shard. It remains approximate across shards; request audit_rows to measure sampled full-graph ID recall. The per-shard pilot accepts distance-equivalent ties with different IDs. CUDA requests never fall back to CPU.

Usage

massive_full_knn_graph(
  x,
  k,
  output,
  backend = "cpu",
  n.cores = 1L,
  chunk_rows = NULL,
  reference_chunk_rows = NULL,
  memory_limit = "8GB",
  checkpoint = FALSE,
  resume = FALSE,
  method = c("exact", "hnsw_sharded", "ivf_sharded", "coarse_postings"),
  audit_rows = 0L,
  checkpoint_every = 100L,
  postings = NULL,
  centers = NULL,
  nprobe = NULL,
  query_order = c("original", "posting"),
  ef_search = 0L
)

Arguments

x

A massive_matrix() descriptor.

k

Number of non-self neighbors, at most 65,536.

output

New prefix for graph files.

backend

"cpu" or "cuda"; CUDA supports exact or IVF shards.

n.cores

CPU workers for each query block.

chunk_rows

Maximum query rows per block; reduced if needed to respect memory_limit. Posting search defaults to at most 65,536 rows per block to reduce repeated posting scans.

reference_chunk_rows

Maximum reference rows per block. For sharded CPU HNSW or CUDA exact/IVF search, the requested maximum resident index-shard size.

memory_limit

Conservative RAM budget.

checkpoint

Save completed query blocks, including sharded search.

resume

Continue a matching checkpoint and partial graph.

method

"exact", "hnsw_sharded", "ivf_sharded", or "coarse_postings". The posting route requires the original file-backed source, postings, and matching centers.

audit_rows

Query rows to compare with streamed exact search (maximum 10,000). Posting search also samples small posting lists and distant-from-center rows; other methods use evenly spaced rows. Defaults to zero. Reported recall covers only sampled rows, not the full graph. sample_target_met requires every sampled row to meet 0.99.

checkpoint_every

Commit every this many query blocks. A shard boundary is always committed. Defaults to 100.

postings

Physical posting descriptor for "coarse_postings".

centers

Centers used to construct postings.

nprobe

Posting lists examined per row; defaults to at most eight.

query_order

For posting search, "original" retains source-row order; "posting" scans grouped posting rows and externally restores original graph order with bounded memory and additional disk space.

CPU HNSW search effort per shard. Zero uses the native shape-based default. A positive value must be at least k + 1 and cannot exceed the smallest reference shard. Other methods reject it.

Value

A validated fastEmbedR_massive_graph descriptor.