chest-ct-foundation-model-benchmark

A single-protocol evaluation of frozen chest-CT foundation models.

Published CT encoders are difficult to compare. Each reports results on its own
data partition, under its own probing recipe, with its own preprocessing. This
repository provides the components needed to compare them directly: a unified
loader that normalizes heterogeneous encoders behind one interface, fixed
patient-disjoint splits distributed as data, and a frozen-probe protocol applied
identically to every model.

Maintained by Evan W. Damron. Apache-2.0. Companion to
DALE-CT.

Contents

• splits/ — the evaluation partitions, as scan identifiers
• results/ — all reported metrics, as CSV
• ctbench/ — the evaluation harness
• configs/ — protocol configurations (embedding extraction, volume-level probes, dense probes)
• tests/ — a synthetic end-to-end gate for the harness

Splits

splits/splits.csv defines every partition used here. Each row assigns one
volume to one role within one experiment:

volume_id,dataset,experiment,role
train_10001_a_1,ct-rate,ctrate_mil,train
valid_3_a_1,ct-rate,ctrate_mil,test
tst00117,rad-chestct,rad_transfer,test

Experiment	Task	Train	Val	Test
ctrate_mil	multi-abnormality classification (CT-RATE)	3,942	1,000	992
dense_probe	ReX / TotalSegmentator coverage	281	47	86
rad_transfer	external transfer (RAD-ChestCT)	2,286	984	360

The CT-RATE partitions are patient-disjoint. Note that CT-RATE assigns patient
identifiers separately within its train and validation pools, so comparing
volume_id.split("_")[1] across the two produces spurious collisions; the
manifests here are derived from the prefixed identifiers and contain no patient
overlap between training and test.

Identifiers are given without file extensions. CT-RATE volumes appear as
{volume_id}.nii.gz in the official release; RAD-ChestCT uses NoteAcc_DEID.

Only identifiers are distributed here — no images and no labels. Obtain
CT-RATE (CC-BY-NC-SA)
and RAD-ChestCT (CC-BY) from their
sources under their respective licenses, and set CTBENCH_DATA_ROOT
accordingly. The label CSVs the probes read are derived from those downloads
by ctbench.prepare_labels (see below).

Preparing the data

RAD-ChestCT is used as distributed: its Zenodo release is already one .npz
per scan, which the loader reads directly.

CT-RATE requires one conversion step. Its NIfTI files do not store Hounsfield
units — each volume carries its own RescaleSlope and RescaleIntercept in the
metadata CSV, and the NIfTI headers leave the corresponding fields unset. The
intercept is not constant across the corpus, so the metadata is required rather
than optional.

python -m ctbench.prepare_ctrate 
    --raw-root $CTBENCH_DATA_ROOT/CT-RATE/dataset/train_valid 
    --metadata $CTBENCH_DATA_ROOT/CT-RATE/dataset/metadata/validation_metadata.csv 
    --out-dir  $CTBENCH_DATA_ROOT/CT-RATE_valid_hu 
    --manifest splits/splits.csv --experiment ctrate_mil

Repeat with train_metadata.csv and an output of CT-RATE_train_hu. The
conversion applies the per-volume rescale, transposes to (Z, H, W), and stores
float16, with no clipping, resampling, or reorientation; model-specific
preprocessing happens later inside each backbone’s extractor. Passing
--manifest restricts the conversion to the volumes the benchmark actually
evaluates. Writes are atomic and existing outputs are validated and skipped, so
an interrupted run resumes cheaply.

Both sources are read by file name, which keeps re-runs cheap: adding a model or
resuming an interrupted extraction touches only the volumes still outstanding. A
sharded-tar reader is retained for the CT-RATE training split under
ctrate_train_webdataset, but it is not the default — that pipeline decodes
before applying the manifest filter, so a resumed run re-reads every volume it
has already processed.

Preparing the labels

The probes read three label CSVs under CTBENCH_DATA_ROOT, none distributed
here. ctbench.prepare_labels builds all three from the official downloads:

python -m ctbench.prepare_labels 
    --ctrate-labels-dir $CTBENCH_DATA_ROOT/CT-RATE/dataset/multi_abnormality_labels 
    --rad-labels-dir    $CTBENCH_DATA_ROOT/RAD-ChestCT

CT-RATE’s train_predicted_labels.csv and valid_predicted_labels.csv are
copied unchanged from the official multi_abnormality_labels directory.
RAD-ChestCT/rad_labels.csv is derived from the Zenodo release’s
img{train,valid,test}_Abnormality_and_Location_Labels.csv: each of the 16
classes shared with CT-RATE is the union of one or more RAD-ChestCT
abnormalities over all location columns (the mapping is in
ctbench/prepare_labels.py and reproduces the paper’s label file exactly).

Results

All metrics are computed on the splits above, with frozen encoders. Full tables
are in results/; the volume-level tables include 95% bootstrap confidence
intervals.

Volume-level classification

Linear multiple-instance probes over per-volume token bags.

Model	CT-RATE AUROC	CT-RATE AUPRC	RAD frozen AUROC	RAD retrained AUROC
COLIPRI-CRM	0.8480	0.5799	0.7469	0.7982
DALE-CT-2S	0.8247	0.5312	0.6252	0.7389
DALE-CT-0-L	0.8156	0.5206	0.6281	0.7572
DALE-CT-1S-v2	0.8098	0.5074	0.6284	0.7334
DALE-CT-0	0.8057	0.5112	0.5946	0.7477
Finetuned DINOv2	0.7953	0.4902	0.6252	0.7550
Merlin	0.7810	0.4631	0.6566	0.7177
TAP-CT-B (2D)	0.7719	0.4623	0.6098	0.7450
CT-FM	0.7670	0.4211	0.5721	0.7273
CT-CLIP	0.6040	0.2564	0.5116	0.5795

Macro averages. CT-RATE test n=992; RAD-ChestCT test n=360. The frozen arm
applies the CT-RATE-trained classifier directly to RAD-ChestCT; the retrained
arm keeps the encoder frozen and fits a new classifier on the RAD-ChestCT
training split. Point estimates come from the validation-selected probe; the
seed-variance stage behind the reported spreads did not run uniformly for the
3D baselines (n_seeds in results/ctrate_indomain.csv: 5 for the 2D models,
4 for COLIPRI-CRM, 1 for the remaining 3D baselines), while the bootstrap CIs
are computed identically for every model.

Dense localization

Linear probes on spatially-resolved features, predicting ReXGroundingCT finding
masks (14 classes) and TotalSegmentator organ coverage (118 classes). Reported
as macro AUROC on the held-out scans; see results/dense_probes.csv for AUPRC
and probe selection details.

Model	Features probed	ReX	TotalSegmentator
TAP-CT-B (2D)	2D patch tokens	0.9283	0.9777
COLIPRI-CRM	volumetric tokens	0.9536	0.8850
Finetuned DINOv2	2D patch tokens	0.9473	0.9644
DALE-CT-2S	2D patch tokens	0.9120	0.9632
DALE-CT-0-L	2D patch tokens	0.9247	0.9607
DALE-CT-0	2D patch tokens	0.8926	0.9546
CT-FM	sliding-window tokens	0.7531	0.9121
DALE-CT-1S-v2	2D patch tokens	0.8733	0.8704
CT-CLIP	volumetric tokens	0.5230	0.5137

Labels are computed against each model’s own token geometry rather than a
shared grid, so every row is spatially aligned by construction: organ coverage
is pooled onto the 24³ token grid for volumetric transformers, computed per
window for sliding-window encoders, and onto the 2D patch grid for slice
encoders.

CT-CLIP’s near-chance result reflects its tokenizer rather than an alignment
error. The token layout was checked against the upstream CTViT implementation,
and the pipeline was validated by an axis-permutation diagnostic on COLIPRI, for
which the assumed layout correlates with anatomy at 0.458 against 0.329 or below
for every alternative ordering.

Pretraining data and domain status

Whether an evaluation is in-domain depends on what each encoder was pretrained
on, so the benchmark records provenance per model rather than labeling the
splits globally. Several encoders were pretrained on the CT-RATE training
split itself; for those, ctrate_mil and dense_probe are in-domain
evaluations, and for Tangerine in particular the probe-training volumes were
seen during pretraining (the evaluation volumes come from the held-out
validation pool). No model in the benchmark saw RAD-ChestCT during
pretraining, which makes rad_transfer a clean external test across the
board.

Model	Pretraining corpus	CT-RATE in pretraining
COLIPRI-CRM	CT-RATE train (volumes + reports)	yes
CT-CLIP	CT-RATE train (volumes + reports)	yes
DALE-CT family	CT-RATE train	yes
Finetuned DINOv2	LVD-142M, then CT-RATE train	yes
Tangerine	98,588 chest CTs across 28 sources; 47% is the CT-RATE train split, plus NLST and the SUMMIT screening cohort	yes (train split only)
TAP-CT-B (2D/2.5D/3D)	104,405 CTs from 19,995 oncological patients, in-house (NKI)	no
Curia / Curia-2	150,000 exams (~72% CT, 28% MR, head-to-toe) from Centre d’Imagerie du Nord, Paris; the two versions share the corpus and differ in recipe	no
Merlin	25,494 abdominal CTs with reports and EHR codes (Stanford)	no
CT-FM	148,000 CTs, mixed anatomy	no
RAD-DINO	~838k chest radiographs (2D)	no
DINOv3	LVD natural images	no

Sources: each model’s paper or model card; McConnell et al. state the CT-RATE
train/validation handling for Tangerine explicitly, and the Curia-2 report
states its pretraining set is shared with Curia.

Model coverage

Encoders expose different structure, and the benchmark records what each one
supports rather than leaving results blank.

Features exposed	Volume-level probe	Dense probe
per-slice CLS and 2D patch tokens	yes	yes, slice and patch level
volumetric or sliding-window token grid	yes	yes, token level
single pooled volume embedding	yes	not applicable

Merlin returns one pooled embedding per volume with no token grid, so dense
probing is not defined for it. Volumetric encoders have no per-slice summary
token, so the slice-level probe applies only to slice-based models.

Installation

git clone https://github.com/Kentucky-Open-Science/chest-ct-foundation-model-benchmark
cd chest-ct-foundation-model-benchmark
pip install -r requirements.txt

Filesystem locations are set through environment variables, each with a local
default:

Variable	Default	Contents
CTBENCH_DATA_ROOT	./data	CT-RATE and RAD-ChestCT volumes and label files
CTBENCH_MODEL_ROOT	./benchmark_models	third-party backbone weights
CTBENCH_CHECKPOINT_ROOT	./checkpoints	local checkpoints
CTBENCH_FEATURES_ROOT	./embeddings	extracted embeddings
CTBENCH_OUTPUT_ROOT	./outputs	probe results

Usage

Evaluation is separated at the embedding boundary, so each encoder is run once
and probes are run as often as needed.

# extract frozen embeddings (once per model x split; --manifest restricts the
# run to the benchmark's fixed splits)
python -m ctbench.embed --config configs/benchmark_embeddings.yaml 
    --model_key dale_ct_2s --split ctrate_train --manifest splits/splits.csv
python -m ctbench.embed --config configs/benchmark_embeddings.yaml 
    --model_key dale_ct_2s --split ctrate_valid --manifest splits/splits.csv
python -m ctbench.embed --config configs/benchmark_embeddings.yaml 
    --model_key dale_ct_2s --split rad --manifest splits/splits.csv

# volume-level probes: select the probe on validation, measure seed variance,
# run the frozen transfer arm, then aggregate with bootstrap CIs
python -m ctbench.run_benchmark --config configs/error_bars.yaml 
    --mode select --task ctrate --model dale_ct_2s
python -m ctbench.run_benchmark --config configs/error_bars.yaml 
    --mode variance --task ctrate --model dale_ct_2s --seed 0
python -m ctbench.run_benchmark --config configs/error_bars.yaml 
    --mode frozen --model dale_ct_2s --seed 0
python -m ctbench.run_benchmark --config configs/error_bars.yaml --mode aggregate

Dense evaluation runs on the 414 ReXGroundingCT-annotated validation scans and
needs its ground truth generated once, then per-model embeddings, then probes:

# token-level ground truth + the annotated-volume list (writes
# $CTBENCH_OUTPUT_ROOT/dense_groundtruth; needs ReXGroundingCT +
# TotalSegmentator masks, see --help for the input paths)
python -m ctbench.dense.groundtruth_2d

# dense embeddings: 2D slice encoders / volumetric encoders
python -m ctbench.dense.extract_2d --config configs/dense_extract_2d.yaml 
    --model_key dale_ct_2s
python -m ctbench.dense.extract_3d --model_key colipri 
    --config configs/benchmark_embeddings.yaml 
    --manifest $CTBENCH_OUTPUT_ROOT/dense_groundtruth/volume_names.json 
    --hu_dir $CTBENCH_DATA_ROOT/CT-RATE_valid_hu 
    --out_root $CTBENCH_FEATURES_ROOT/dense_3d

# dense probes (one config per model group; see configs/)
python -m ctbench.dense.probe --config configs/dense_probe_2d_ts.yaml
python -m ctbench.dense.probe --config configs/dense_probe_colipri_ts.yaml

Volume-level protocol: a grid over six learning rates and three pooling schemes
(average, maximum, learned attention), a linear layer over the label set, probe
selection by validation AUPRC, per-class F1-maximizing thresholds fit on the
validation split, and 2,000-resample bootstrap confidence intervals. Resample
indices are fixed per task and shared across models, so paired comparisons
remain valid.

Dense protocol: a linear layer on frozen tokens, trained for 15,000 steps with
SGD, learning rate selected from a four-point grid by validation AUPRC.

Testing

tests/smoke_synthetic.py runs the volume-level chain (select, variance,
frozen transfer, bootstrap aggregate) and both dense probes end-to-end on
synthetic embedding bags with a planted linear signal — no datasets or model
weights required, a few minutes on CPU or GPU:

python tests/smoke_synthetic.py

Adding a model

ctbench/backbones/loader.py is the entry point. It absorbs each encoder’s
input contract — Hounsfield windowing, resampling, cropping and padding,
foreground selection, and token flattening — behind one interface.

Register a loader that returns the model together with its specification, and
provide an extraction function that emits either a per-slice CLS bag or a token
grid. Everything downstream is model-agnostic. Dense evaluation additionally
requires labels at the new model’s token geometry; ctbench/dense/ contains
generators for slice, volumetric, and sliding-window layouts.

Citation

@article{damron2026dalect,
  title  = {DALE-CT: Depth-Aware 2D Slice Encoders Learn an Anatomical
            World Model of Chest CT},
  author = {Damron, Evan W. and Gokmen, Mahmut S. and Klusty, Mitchell A. and
            Leach, Caroline N. and Collier, Emily B. and Bumgardner, V. K. Cody},
  year   = {2026}
}

Repository metadata for citation is in CITATION.cff.

---

Imported from gh:Kentucky-Open-Science/chest-ct-foundation-model-benchmark. Source last updated 2026-08-26. Synced 2026-08-26.