Experimental file-backed landmark embedding projection — massive_project

Projects each saved query-to-landmark KNN batch into a fixed reference embedding. Fit umap() or tsne() on the same landmark rows used to construct graph. Only the query coordinates are written; reference coordinates are not moved. Query batches are independent, so changing chunk_rows can change stochastic refinement results.

Usage

massive_project_landmarks(
  fit,
  graph,
  output,
  selection = NULL,
  backend = "cpu",
  n.cores = 1L,
  chunk_rows = NULL,
  memory_limit = "8GB",
  refinement_epochs = 50L,
  transform_iter = 250L,
  transform_perplexity = 5,
  seed = 4L,
  checkpoint = FALSE,
  resume = FALSE,
  source = NULL,
  local_refine = FALSE,
  local_neighbors = 15L,
  overlap_rows = 500L,
  row_range = NULL,
  devices = NULL
)

Arguments

fit: A fastEmbedR_embedding fitted on the landmark reference.
graph: File-backed query-to-landmark KNN from massive_landmark_knn().
output: New row-major float32 .f32 output path.
selection: Optional result of massive_select_landmarks() from the same query source. Its rows retain the fitted reference coordinates.
backend: "cpu" or "cuda". CUDA supports zero-epoch UMAP projection and 2D or 3D t-SNE fixed-reference transformation. Unsupported CUDA routes fail without CPU fallback.
n.cores: Number of CPU workers.
chunk_rows: Maximum query rows per projection batch.
memory_limit: Conservative RAM budget.
refinement_epochs: UMAP fixed-reference refinement epochs; zero uses weighted KNN projection only.
transform_iter: t-SNE fixed-reference transform iterations.
transform_perplexity: t-SNE transform perplexity, at most graph k.
seed: Random seed for batch-specific optimizer streams.
checkpoint: Save completed row counts after each batch. With devices, also retain a parent manifest for shard recovery.
resume: Continue from matching checkpoint and partial output. A missing .part file is allowed only before the first row is written; completed CUDA shards are reused before the final merge.
source: Query massive_matrix() descriptor. Required only when local_refine = TRUE; it must match the saved KNN source identity.
local_refine: Reconstruct a bounded CPU query-to-query graph in overlapping windows before UMAP refinement. EXPERIMENTAL and opt-in.
local_neighbors: Non-self query neighbors per local window.
overlap_rows: Extra query rows on each side of a window; must be at least local_neighbors.
row_range: Optional inclusive query-row range, used for independent projection shards. Output contains only that range.
devices: Optional distinct zero-based CUDA device indices. Only independent projection batches are distributed; reference fitting and KNN construction remain on one device. Shards remain on disk.

Value

An experimental file-backed result with layout as a fastEmbedR_massive_matrix descriptor. A .manifest.rds sidecar supports massive_open_projection() on the same files.