SmolVLM-256M on Core ML — with a dependency-free Swift CLI

Run a multimodal VLM (image → caption / VQA) entirely on Core ML, called from a zero-dependency Swift CLI whose preprocessing is bit-exact with the HuggingFace reference (custom libjpeg-compatible JPEG decoder and Pillow-exact LANCZOS resampler, both written in Swift and verified against golden data on every run).

image ──► vision (Core ML, fp16) ──► prefill (Core ML, K/V as plain outputs)
                                         │
text ──► BPE tokenizer (Swift) ──────────┴─► decode (Core ML, S=1, MLState) ──► tokens

Measured on an Apple Silicon Mac (macOS 26.6.2, fp16, warm process, with the default --compute-units auto selecting .all): caption in ~1.3 s end-to-end — vision ~87 ms, 81-token prefill + first logits ~200 ms, decode ~43 tok/s. On the conservative .cpuAndGPU setting vision is ~167 ms and decode ~40 tok/s.

📄 Full write-up: docs/field-report.md — every crash, the workaround architecture, and the measured constraints, in long form.

Converting the model was the easy part. The valuable part is what we learned getting generation to run reliably, none of which is documented anywhere:

Most of this table was fixed by a macOS point release. The findings below were measured on macOS 26.5.1. This Mac then updated to 26.6.2 and, re-tested against the real packages, findings 1, 2 and 5 no longer reproduce; 3 still crashes; 6's Python .mlpackage + ALL half still crashes while the compiled path works; 7 was confirmed by the update itself. The table at the top of docs/field-report.md has the current verdicts, and python revalidate.py regenerates them on your own machine. Read the rows below as what a stateful Core ML pipeline can do to you, not as a list of live bugs.

The architecture is the workaround: prefill is stateless and returns all 30 layers' K/V as plain outputs [1,90,P,64]; decode takes them back as plain inputs each step and keeps state only for generated tokens. No state-transfer API is ever crossed. This is why the same design ported to Swift without a single crash.

This GitHub repo holds code only — no weights, no goldens. Everything large or generated lives in three Hugging Face repos, split by what you need it for:

The Space does not run Core ML, and never will. Hugging Face Spaces are Linux; Core ML executes only on Apple platforms. That is not a limitation to work around — it is the reason the Space is useful. It is the one piece of this project whose output does not move when your Mac's OS moves, so it stays a fixed reference point for finding 7 (greedy decode output drifted across OS sessions on macOS 26.5, with bit-identical inputs verified at every stage). Use it to answer "did my machine change, or did I break something?"

Both HF repos mirror this checkout's directory layout — packages at the top level, runtime and golden assets under exports/ — so --local-dir . from the cloned repo root drops every file exactly where the CLI's single --root . already looks for it.

git clone https://github.com/NagaYu/smolvlm-coreml && cd smolvlm-coreml

# RUN: the minimal fp16 set — three packages + tokenizer/embeddings (model repo)
huggingface-cli download YutaN13/SmolVLM-256M-Instruct-CoreML --local-dir . \
  --include "smolvlm_vision.mlpackage/*" \
            "smolvlm_prefill_kv.mlpackage/*" \
            "smolvlm_decode_hybrid.mlpackage/*" \
            "exports/embeddings.bin" "exports/tok/*" "exports/specials.json"

# VERIFY: goldens for --check (dataset repo — note --repo-type dataset)
huggingface-cli download YutaN13/smolvlm-coreml-golden --repo-type dataset --local-dir . \
  --include "test_image.jpg" \
            "exports/golden.json" "exports/golden_pixel_values.bin" "exports/pil_decode.npy"

cd SmolVLMSwift && swift build -c release && cd ..
./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --root .
./SmolVLMSwift/.build/release/smolvlm --check --root .

Add-ons, each optional:

• 13-tile hi-res split path: add --include "*_1024*" to the model download, and "exports/golden_split.json" "exports/golden_pixel_values_split.bin" to the dataset download (that pair is 39 MB of the dataset's 43 MB — skip it if you never pass --split).
• INT8 prefill/decode: add --include "*_int8*". They are roughly half the size and cost ~12 s of GPU shader compilation per process (macOS 26.5 measurement); the CLI defaults to fp16.
• Prefer to build everything locally? python convert_all.py && python export_artifacts.py reproduces both repos' contents from source. The HF repos exist to save you ~10 minutes and a torch install, not to hide anything.

Everything published in those repos is stamped to the environment it was produced on: macOS 26.5 (Darwin 25.5.0), Apple silicon arm64, coremltools 9.0, Python 3.13, Swift 6.3. If you are on a newer OS, read docs/os-upgrade.md before trusting the goldens: run python revalidate.py first.

# 1. Convert models (~10 min, downloads SmolVLM-256M; needs torch + coremltools 9)
pip install torch transformers coremltools pillow
python convert_all.py            # or --fast for the minimal fp16 set

# 2. Export tokenizer/embeddings/golden test data
python export_artifacts.py

# 3. Build and run the Swift CLI (macOS 15+, Swift 6)
cd SmolVLMSwift && swift build -c release && cd ..
./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --root .
./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --question "How many cats?" --root .
./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --split --root .   # 13-tile hi-res
./SmolVLMSwift/.build/release/smolvlm --check --root .                          # verify bit-exactness

--check re-proves on every run: tokenizer ids match HF exactly (81/81 and 876/876), the Swift JPEG decoder matches libjpeg at 0/255, the Swift resampler matches Pillow at maxdiff 0.000000, and full preprocessing matches the golden tensors to fp16 precision. See SmolVLMSwift/README.md for the per-stage parity table.

Model weights (*.mlpackage) and large binaries are not committed — regenerate them with the two scripts above, or download the converted packages from the release page (if published) — see scripts/upload_models_to_hf.py.

• Findings are environment-stamped (macOS 26.5, coremltools 9.0, Apple Silicon). OS updates may fix the crashes — or change greedy outputs (we measured both).
• This is a reference implementation for one model, not a library. Dims (576/30 layers) are hardcoded where it keeps the code readable.
• The bundled JPEG decoder covers baseline 4:4:4 and grayscale (what PIL/libjpeg parity was verified against); other JPEGs fall back to CoreGraphics (±4/255 decode variance).
• INT8 packages pay a ~12 s GPU shader compile per process; fp16 does not. The CLI defaults to fp16.
• Split-path (--split) captions can differ from a torch-fp32-vision HF run in late tokens: running vision in fp16 is inherent to on-device inference, not a porting bug (the Python Core ML pipeline produces token-identical output to Swift).

• Code: MIT (see LICENSE).
• Model: HuggingFaceTB/SmolVLM-256M-Instruct, Apache 2.0. Converted .mlpackage files are derivative works of those weights.
• Test image: "Tabby cat with blue eyes" via Wikimedia Commons, CC0 (re-encoded to 640×480 baseline 4:4:4).