Reusable Vision Representations

Tools and reproducible experiments for measuring selectivity, cross-context portability, and compositional transfer in frozen visual representations. The repository provides dataset preparation, frozen feature extraction, discrete-token diagnostics, held-out-composition probes, a prospectively frozen representation-ranking score, compact numerical results, and regression checks.

• Selectivity (S_TOKEN): how strongly discrete patch-token distributions separate generic parts.
• Portability (P_CMI): how little information a token distribution retains about the object context once the generic part is fixed.
• Compositional transfer: part recognition on part-by-object-class edges excluded from probe training, compared with evaluation on seen edges.

The diagnostics are kept separate. A representation can be stable across contexts without distinguishing parts, or selective while encoding a part in a context-specific way. See the mathematical definitions.

Across the evaluated models and domains, specialization often increased through depth while portability decreased. Unseen-composition accuracy frequently peaked before the final representation. The effect is not universal: the public results retain weak, positive, and otherwise nonconforming trajectories alongside the recurring pattern.

The fixed PartImageNet graphs contain 4 parts, 46 classes, and 146 edges for Quadruped; 5, 14, and 51 for Bird; and 5, 17, and 73 for Biped. Detailed results, including counterexamples, are in docs/results.md.

python -m venv .venv
python -m pip install --upgrade pip
python -m pip install -e ".[test]"
pytest -q
python scripts/verify_release.py --repo-root .
python scripts/generate_figures.py --repo-root . --output-root outputs/figures

These commands use only the compact committed numerical artifacts. They do not download a dataset or model.

Obtain the official PACO annotations and required COCO images first. The commands below keep all dataset content and generated arrays outside Git.

python -m pip install -e ".[data,gpu]"
python scripts/prepare_paco.py \
  --data-root /path/to/datasets \
  --output-root outputs/paco/prepared

python scripts/extract_features.py \
  --metadata outputs/paco/prepared/sample_metadata.csv \
  --model dinov2_vitb14 \
  --model-registry configs/models/registry.yaml \
  --cache-root /path/to/model-cache \
  --output-root outputs/paco/features \
  --image-column image_path --mask-column mask_path

python scripts/compute_diagnostics.py \
  --dataset paco \
  --metadata outputs/paco/prepared/sample_metadata.csv \
  --graph-root results/paco/balanced_graph.csv \
  --feature-root outputs/paco/features \
  --model dinov2_vitb14 \
  --output-root outputs/paco/diagnostics

python scripts/run_compositional_eval.py \
  --metadata outputs/paco/prepared/sample_metadata.csv \
  --features outputs/paco/features/dinov2_vitb14/part_pooled.npy \
  --model dinov2_vitb14 \
  --default-domain PACO \
  --output-root outputs/paco/evaluation

Repeat extraction and evaluation for the model names in configs/paco/default.yaml. Annotation processing is resumable with --candidate-table. A previously generated public-compatible feature cache can be verified end to end:

python scripts/reproduce_paco.py \
  --stage cached \
  --metadata /path/to/paco/sample_metadata.parquet \
  --cache-root /path/to/paco/feature-cache \
  --output-root outputs/paco/regression

The external protocol pools the official segmentation train and validation images, assigns whole images to a global deterministic 70/30 split with SHA-256 and seed 424242, requires at least 30 training and 10 evaluation samples per retained edge, and applies the fixed sparse degree core. The redesign was completed from annotation counts before any external model inference.

python -m pip install -e ".[data,gpu]"
python scripts/prepare_partimagenet.py \
  --data-root /path/to/PartImageNet_Seg \
  --output-root outputs/partimagenet/prepared

python scripts/extract_features.py \
  --metadata outputs/partimagenet/prepared/sample_metadata.csv \
  --model dinov2_vitb14 \
  --model-registry configs/models/registry.yaml \
  --cache-root /path/to/model-cache \
  --output-root outputs/partimagenet/features

python scripts/compute_diagnostics.py \
  --metadata outputs/partimagenet/prepared/sample_metadata.csv \
  --graph-root results/partimagenet/graphs \
  --feature-root outputs/partimagenet/features \
  --model dinov2_vitb14 \
  --output-root outputs/partimagenet/diagnostics

python scripts/run_compositional_eval.py \
  --metadata outputs/partimagenet/prepared/sample_metadata.csv \
  --features outputs/partimagenet/features/dinov2_vitb14/part_pooled.npy \
  --model dinov2_vitb14 \
  --output-root outputs/partimagenet/evaluation

To validate all five ViT trajectories from an existing compatible local feature cache:

python scripts/reproduce_partimagenet.py \
  --stage cached \
  --metadata /path/to/partimagenet/sample_metadata.parquet \
  --cache-root /path/to/partimagenet/feature-cache \
  --output-root outputs/partimagenet/regression

All path arguments are user-controlled. Interrupted feature extraction reuses a completed model cache and never requires manual CSV edits.

The primary ViT trajectories include DINOv2 ViT-B/14, OpenAI CLIP ViT-B/16, DINOv3 ViT-B/16, SigLIP 2 Base/16, and supervised AugReg ViT-B/16. PACO also includes MAE ViT-B/16 and OpenCLIP ViT-B/16 trained on LAION-2B. The secondary architecture check uses DINOv3 and supervised ConvNeXt-Base. Exact identifiers, expected weight hashes, normalization, patch sizes, final-normalization rules, and special-token exclusions are in configs/models/registry.yaml and docs/models.md.

No pretrained weight is included.

The recorded runs used an NVIDIA GeForce RTX 5070 Laptop GPU with about 8 GB VRAM, Python 3.10.11, PyTorch 2.11.0 with CUDA 12.8, and frozen inference only. At batch size 8, recorded PyTorch peak allocation was about 0.58–0.72 GB for the ViT extractors and about 0.51 GB for ConvNeXt; these figures do not include every driver-level allocation. Full cached intermediates occupied about 15.8 GB for PACO and 1.9 GB for PartImageNet, excluding datasets and provider caches. See docs/compute.md.

• src/reusable_vision_representations/: reusable split, graph, pooling, diagnostic, probe, ranking, statistics, and reporting code
• scripts/: preparation, extraction, evaluation, reproduction, and verification entry points
• protocols/: fixed scientific definitions and seeds
• configs/: runtime and model registry configuration
• results/: compact machine-readable outputs and hashes
• figures/aggregate/: plots derived only from numerical results
• docs/: methods, data, models, history, findings, compute, and limitations
• tests/: CPU-only checks used by continuous integration

Datasets, annotations, images, and pretrained weights are not distributed here and are not covered by the repository's MIT license. Obtain them from their official sources and follow their terms. See docs/datasets.md, docs/models.md, and THIRD_PARTY_NOTICES.md.

The evidence covers a finite set of models, datasets, parts, object classes, and interventions. The ranking score orders representations; it is not a calibrated accuracy estimate. Spatial resolution changes confound ConvNeXt stage depth, and several external trajectories do not show the recurring selectivity-portability pattern. See docs/limitations.md.

Cite the software repository using CITATION.cff.