Skip to contents

EXPERIMENTAL massive-data community clustering

Source: R/massive_cluster_workflow.R
massive_cluster.Rd

Selects bounded resident landmarks from a file-backed matrix, builds their graph with knn_graph(), and runs the existing native Walktrap, Louvain, or Leiden implementation. Query-to-landmark neighbors and weighted-vote labels remain file-backed. This is a landmark approximation, not clustering of the complete graph. With a symmetric file-backed fuzzy graph and massive = "out_of_core_graph", the function runs CPU multilevel Louvain or Leiden on the file-backed graph. Full-graph Leiden runs one hierarchy with n_iterations local-move passes per level. Full-graph Walktrap runs exact walks within bounded blocks and contracts communities; it is an explicit approximation when more than one block is needed. massive = "auto" uses ordinary in-memory graph clustering when resource estimates permit it. Exact Walktrap is CPU-only and limited to 4,000 landmark vertices; automatic planning includes its quadratic transition-matrix memory.

Usage

massive_cluster(
  x,
  method = c("leiden", "louvain", "walktrap"),
  massive = c("landmark", "auto", "out_of_core_graph"),
  landmarks = NULL,
  k = 30L,
  output = NULL,
  backend = "cpu",
  n.cores = 1L,
  devices = NULL,
  landmark_method = c("reservoir", "random"),
  knn_method = c("auto", "exact", "hnsw", "ivf"),
  chunk_rows = NULL,
  memory_limit = "8GB",
  resolution = 1,
  n_iterations = 10L,
  n_runs = 1L,
  steps = 4L,
  seed = 1L,
  checkpoint = FALSE,
  resume = FALSE,
  checkpoint_every = 100L,
  initial = NULL,
  nn = NULL
)

Arguments

x

A massive_matrix() descriptor, a symmetric file-backed fuzzy graph in full-graph mode, or a resident matrix in auto mode.

method

"leiden", "louvain", or "walktrap" (random walks).

massive

"landmark", "auto", or "out_of_core_graph".

landmarks

Landmark count; auto mode chooses a bounded count if NULL.

k

Number of non-self neighbors in the landmark graph and projection.

output

New prefix for landmark files, or a new .clusters.u32 label-file path in full-graph mode.

backend

"cpu" or "cuda"; full-graph clustering is CPU-only.

n.cores

CPU worker count.

devices

Optional CUDA device indices for landmark clustering. The first builds the reference graph; all search disjoint query shards.

landmark_method

"reservoir" or "random" landmark selection.

knn_method

Query-to-landmark nearest-neighbor search method.

chunk_rows

Maximum source rows per streamed batch.

memory_limit

Conservative RAM budget; CUDA also checks free VRAM.

resolution, n_iterations, n_runs, steps, seed

Graph clustering controls.

checkpoint, resume

Save or reuse completed landmark workflow stages, label batches, or completed full-graph Louvain/Leiden stages. Full-graph Walktrap does not yet support checkpoints.

checkpoint_every

Landmark label batches between checkpoints.

initial

Optional saved landmark-clustering result used to initialize full-graph Louvain or Leiden. Other modes reject it.

nn

Previous massive landmark UMAP or t-SNE result whose verified selection and query KNN should be reused instead of recomputed.

Value

An experimental file-backed clustering result, a file-backed Walktrap, Louvain, or Leiden result in full-graph mode, or an in-memory graph_cluster() result when auto mode selects the resident route.