Skip to contents

FiberMargin Benchmarks

Source: vignettes/benchmarks.Rmd
benchmarks.Rmd

Evaluation contract

benchmark_spatial_refiners() evaluates each method on identical inputs and the same reference data:

  • coordinates (xy)
  • initial labels (labels)
  • optional specimen ids (samples)
  • optional boundary/sparse annotations used for stratified scores

It reports elapsed time and a standard metric table generated by evaluate_spatial_refinement().

Standard metrics reported

  • accuracy, initial_accuracy, accuracy_gain
  • correction_recall and damage_rate
  • worst_recall, macro_recall
  • boundary_accuracy, sparse_region_accuracy
  • ari

Dataset catalog 

available_spatial_benchmarks()
#>   dataset included observations classes scenarios
#> 1   dlpfc     TRUE        47329       7        45
#> 2 merfish    FALSE        28317       8        45
#> 3     crc     TRUE       194541      19        60
#>                                                                  license
#> 1                                Artistic-2.0 (spatialLIBD data package)
#> 2 CC0 raw Dryad data; no explicit license for derived BASS domain labels
#> 3         CC BY 4.0 (10x Genomics source and author-derived annotations)
#>                                                                                               source
#> 1                                                      https://bioconductor.org/packages/spatialLIBD
#> 2                                                              https://doi.org/10.5061/dryad.8t8s248
#> 3 https://www.10xgenomics.com/datasets/visium-hd-cytassist-gene-expression-libraries-of-human-crc-v4
#>                                                                                                                               note
#> 1                                                             Coordinates, layer labels, and frozen corruption inputs are bundled.
#> 2                                                               Use the publication script with a local processed annotation file.
#> 3 Coordinates, 19 WSI annotation labels, and deterministic corruption recipes are bundled; counts and tissue imagery are excluded.

Load one real scenario 

dlpfc <- load_spatial_benchmark("dlpfc", scenario = 1L)
names(dlpfc)
#>  [1] "xy"                     "labels"                 "truth"                 
#>  [4] "samples"                "boundary"               "regions"               
#>  [7] "sparse"                 "name"                   "metadata"              
#> [10] "spot_id"                "subject"                "nearest_adjacent_layer"
#> [13] "scenario"

Example benchmark run 

bench <- simulate_spatial_domains(
  n = 5000L,
  pattern = "jagged_stripes",
  noise = 0.20,
  samples = 2L,
  seed = 7L
)

benchmark_spatial_refiners(
  data = bench,
  methods = list(
    FiberMargin = refine_spatial_labels,
    InitialOnly = function(xy, labels, ...) labels
  ),
  include_initial = TRUE,
  seed = 1L
)

Reproducibility scripts 

Rscript benchmarks/run_publication_benchmarks.R
Rscript benchmarks/run_fibermargin_publication_benchmarks.R
Rscript benchmarks/run_general_mask_repair_benchmarks.R

Each script stores results and artifacts with fixed random seeds so runs are reproducible.