Status: Living document. Version 0.3 (2026-04-17). Audience: Engineers at Anthropic and elsewhere who want to verify (or refute) the claims in this repo's README. Position: We will state what we have measured, what we have estimated, and what is unmeasured. Where a number is weak we say so.

• The headline numbers in the README (32% tokens, 18.8% cost) come from a single 2-file fixture with N=1. They are illustrative, not representative. Treat them as an existence proof that Context OS does reduce tokens on at least one realistic prompt, nothing more.
• The only numbers we consider scientifically defensible today are (a) the reducer benchmarks on logs/JSON/config fixtures (N=7 cases, paired before/after, deterministic transform — see §7.1) and (b) the compaction-survival benchmark (N=2 cases — see §7.2). Both are offline, deterministic, and have well-defined gates.
• Everything else in the README (noise filtering K-token estimates, 40–65% output reduction, Haiku 93% exploration savings, 27–70% test-run compression) is either: cited from upstream work, estimated from documented Anthropic behavior, or pending proper measurement. §8 tags each of the 16 techniques with its evidence class.
• Real savings are distributional, not a point estimate. We report medians and interquartile ranges, not means. We will not quote an unqualified "X% saved" without specifying fixture, task, model, cache state, and N.
• If you run our benchmark once on one repo and get a different number than the README, the README is the one at fault for being under-specified, not your run.

A "token savings" claim for a Claude Code optimization kit has at least seven free variables:

Any benchmark that pins all seven and reports a single number is measuring a point on a seven-dimensional surface. The README's 32% is exactly that: one point, reported for transparency, not a population estimate.

This document defines the measurement protocol needed to produce numbers that generalize. We are publishing the protocol before we have the full corpus; §9 lists what's still missing.

We distinguish four levels of measurement. Every claim in the repo should be tagged with which level it lives at.

A session = one invocation of claude (interactive or --print) from start to stop. A session emits a final JSON usage block from Claude Code containing:

• input_tokens — fresh input this turn.
• cache_read_input_tokens — input served from prompt cache.
• cache_creation_input_tokens — input written into cache this turn.
• output_tokens — tokens emitted by the model.

We define:

total_tokens = input_tokens + cache_read_input_tokens + cache_creation_input_tokens + output_tokens

This is the primary metric. It is cache-aware (cached reads count as tokens processed) but not cost-weighted — for cost see §2.4.

Source: Claude Code --output-format json returns these fields directly (Claude Code docs, headless mode). The benchmark script in scripts/benchmark.sh extracts them via jq.

A task = one user prompt until the model returns stop_reason ∈ {"end_turn", "stop_sequence"} without a tool call triggering a continuation. Tasks are sub-units of sessions. A 40-turn coding session is ~40 tasks.

Per-task measurement requires streaming the transcript and segmenting. Claude Code's --output-format stream-json provides turn-level events (Claude Code docs, SDK streaming). We have not built this aggregator yet (see §9).

For each of the 16 techniques (see §8), the contribution is measured by leave-one-out ablation: run the full Context OS install minus technique i, vs the full install, on the same fixture and task. The delta is technique i's contribution conditional on all others being active.

Caveats:

• Techniques interact. Response shaping + output compression partially overlap. The marginal contribution of each depends on the order and the baseline.
• Some techniques (e.g. statusLine) have structurally zero token effect; measuring them produces noise around zero. We flag these as "non-token UX techniques" in §8.

We have not yet run ablation at scale. The per-technique numbers cited in README column 4 are upstream citations or estimates; see §8 "evidence class" column.

Claude Code reports total_cost_usd per session in the --output-format json response. We use this directly when available.

For manual accounting we use Anthropic's published Sonnet 4.6 pricing (anthropic.com/pricing, retrieved 2026-04-17):

"Blended" pricing for back-of-envelope estimates uses a session-weighted average. We assume a typical mix of 70% cache-read / 20% output / 10% fresh-input, giving ≈$3.41/M. This is the number behind the README's $0.40/session estimate. It is an estimate, not a measurement, and is only used in the --measure dry-run output.

Dollar deltas track token deltas monotonically but not linearly; a session that trades cache reads for fresh input can reduce tokens and increase cost.

We define four synthetic size classes. Each is a reproducible seed of a real-world repo shape.

Current status: only the tiny fixture has been benchmarked end-to-end (README §Measured results). Small/medium/large are pending (see §9).

Reproducibility: each fixture should be a git commit with a pinned SHA, so git clone && git checkout <sha> yields bit-identical state. The current tiny fixture at /tmp/cos-bench-test is not pinned — it's generated ad hoc. This is a bug. See §9.

Tasks must be:

1. Deterministic in intent — unambiguous enough that Claude's variance is in wording, not in action.
2. Realistic — drawn from real user sessions, not synthetic.
3. Bounded — one task should complete in <60s wall clock with Sonnet.

We have three task families planned, one implemented:

The repo-map task was chosen first because it exercises noise filtering and repo-map hints directly. It does not exercise response shaping (low output) or output compression (no test runs). The 32% figure is heavily tilted toward filesystem-scan savings and under-reports the full stack.

• Minimum N: 5 runs per (fixture × task × condition) cell.
• Summary statistic: median, not mean. Claude's output is heavy-tailed — a single run that triggered extended thinking can inflate a 5-run mean by 40%.
• Variance measure: interquartile range (IQR = Q3 − Q1). Quoted as median [Q1–Q3].
• Comparison: paired design — same task run in baseline and treatment, in randomized order, using a fresh temp clone each time so there's no carryover.
• Report format: total_tokens = 54,036 [48,200–58,900] (median and IQR across 5 paired runs).

The current headline number has N=1. We have no variance estimate. This is the single biggest gap in our evidence.

Two canonical conditions must be reported separately:

• Cold cache: first invocation after ≥1h idle, or after claude --resume with a fresh session ID. Cache reads ≈ 0.
• Warm cache: second invocation within 5 minutes of an identical prior run. Cache reads ≈ full system prompt.

The README's 32% is a warm-cache number (74K cache reads before, 48K after). The cold-cache number will be much smaller in absolute terms but potentially larger in percentage, because noise filtering reduces the content that would be cached.

Cold-cache runs are pending (§9).

For each (fixture, task, cache-state, N) cell:

1. Clone fixture into two temp dirs: before/ and after/.
2. Run Context OS setup on after/ only.
3. Interleave runs: before[0], after[0], before[1], after[1], ... to minimize time-of-day cache drift.
4. Record raw JSON transcripts to disk (not just aggregated numbers).
5. Compute paired deltas: one Δtokens per run pair, then take median over pairs.

This is what scripts/benchmark.sh implements at N=1. Extending to N=5 is a matter of wrapping the current script — tracked in §9.

Claude's outputs are non-deterministic even at temperature=0 (we do not set it, so default applies). Output token count varies turn-to-turn for an identical prompt. Published studies (Anthropic, "Inference and reproducibility") suggest output-token coefficient of variation ≈ 5–15% for short deterministic prompts, higher for open-ended tasks.

Implication: Any headline claim below ~10% reduction on a single run is indistinguishable from noise. The 32% figure is above this floor but we still lack a confidence interval.

Prompt cache entries expire after 5 minutes (default) or 1 hour (opt-in, ENABLE_PROMPT_CACHING_1H=1). Two runs 7 minutes apart, default TTL, will both be cold. Two runs 7 minutes apart, 1h TTL, will both be warm. We never compare across TTL settings; TTL is treated as part of the condition.

Absolute cache_read numbers drift over the course of a day because other users' traffic affects cache availability (cache is per-workspace but subject to backend eviction; Anthropic does not publish eviction policy). We observe run-to-run drift of 10–20% on the same task under nominally identical conditions.

Four techniques only show savings under specific conditions:

• Prompt caching 1h — no effect on single-session tasks; large effect on multi-session work over <1h.
• Early compaction (80%) — no effect on sessions that never hit 80% context.
• Context cap (150K) — no effect on sessions that stay below 150K anyway.
• Session memory — no effect within a session; benefit is amortized across session restarts.

Any benchmark with a single short session under-reports the value of these techniques by design. Our tiny-fixture 32% does not reflect them at all.

Two techniques have structurally zero direct token effect:

• statusLine — writes to a shell-visible status bar, not the context window.
• Secret exclusion (security subset of .claudeignore) — tokens are counted against the broader noise-filter entry; secret filtering is a correctness/safety property, not a token saving.

Both are included in the installer because users ask for them. We do not claim token savings for them.

Context OS injects ~300 tokens of CLAUDE.md content and ~150 tokens of settings metadata. On a single short task this is visible in the "before" / "after" cache_creation_input_tokens delta (+301 in the tiny fixture: 5255 → 5556). The overhead amortizes across turns — by turn 2 or 3 the savings dominate on any non-trivial session. But on a truly tiny session (1 turn, 1 file) Context OS can cost more than it saves. We do not hide this.

# Requires: claude CLI on PATH, git, jq.
git clone https://github.com/sravan27/context-os && cd context-os
scripts/benchmark.sh /path/to/target/repo --model sonnet
# Output: JSON report at /tmp/cos-last-benchmark.json

This reproduces the README's headline number when pointed at a tiny fixture. It will produce different numbers on different repos — that is the point of §1.

python3 python/evals/runners/safe_mode_runner.py
# Output: python/evals/reports/safe-mode-report.{json,md}

Fully deterministic. Same input → same output. N=7 cases, each paired before_tokens vs after_tokens on the same raw log fixture. No Claude API calls.

Latest run: 2026-04-10T16:29:14Z. Mean reduction 27.3% (range 13.0–42.3%). See python/evals/reports/safe-mode-report.md.

python3 python/evals/runners/compaction_survival_runner.py
# Output: python/evals/reports/compaction-survival-report.{json,md}

Deterministic. N=2 cases. Verifies that the resume-packet generator retains decisions, modified files, next-step pointers, and stays within a token budget under synthetic noise.

Latest run: 2026-04-10T16:29:16Z. 2/2 pass. Mean packet size 184 tokens.

curl -fsSL .../setup.sh | bash -s -- --measure

This is not a measurement. It is a formula (setup.sh:42–100) applied to file counts. It produces the README's ~103K tokens/session figure. The constants (NOISE_PER_SEARCH=10000, RESPONSE_PER_SESSION=20000, etc.) are educated guesses documented inline. They have not been validated against real sessions. The number should be read as "this is what we'd estimate under these assumptions," not "this is what you'll save."

We have flagged this in the README ("Conservative per-session savings") but the word "conservative" is load-bearing in a way that should not be taken on faith. Validation against a corpus of real sessions is pending (§9).

All dollar figures in this repo use Sonnet 4.6 pricing as of 2026-04-17:

Source: anthropic.com/pricing. These figures change without notice. Any dollar quote older than the pricing page should be recomputed before being cited.

We do not convert Claude Pro / Max plan "5-hour window" savings to dollars — usage-cap headroom is not fungible with dollars. We only quote Pro/Max savings in qualitative terms ("longer sessions before hitting the cap"). README does this correctly.

Source: python/evals/reports/safe-mode-report.json, run 2026-04-10T16:29:14Z.

• Mean reduction: 27.3%. Range: 13.0–42.3%.
• 6/6 reducer cases pass the recall gate (100% of protected strings preserved).
• 1/1 prompt-lint case passes the finding-recall gate.
• N=6 cases, but each case is a single deterministic transform — no variance to report.

Scope: this measures the reducer (the hook that compresses tool output before Claude sees it). It does not measure downstream session-level savings, because Claude would react differently to the compressed output in a real session. We consider this a lower bound on the reducer's contribution but cannot yet quantify the session-level multiplier.

Source: python/evals/reports/compaction-survival-report.json, run 2026-04-10T16:29:16Z.

• 2/2 cases pass.
• Packet size within budget on both.
• Failed-approach retention 1.00 and 0.00 respectively — the second case deliberately tests graceful degradation under noise; we retain pinned facts and drop failed approaches under budget pressure. Passing gate is "pinned / current-subtask / latest-decision / modified-files retention = 1.0". Failed-approach retention is not a gate.

This is not a token-savings benchmark. It's a correctness benchmark for the session-memory technique (row 16 in the README's techniques table). Token savings from session memory come from not re-explaining after a restart — unmeasured (§9).

Source: python/evals/reports/autocontext-eval.md (precision/recall/MRR + baseline lift), python/evals/reports/session-replay.md (token-savings simulation). Runners: python3 python/evals/runners/autocontext_eval.py --baseline naive-filename, python3 python/evals/runners/session_replay.py. Both regenerated on every main CI run; artifacts uploaded for 30 days.

Fixtures (3 parallel mini web-apps, one per language):

• python/evals/autocontext_fixture/ — 13 .py source files + 5 importer-edge tests, ~30 top-level symbols
• python/evals/autocontext_fixture_js/ — 13 .ts files covering the same auth/api/config/db/utils layout
• python/evals/autocontext_fixture_rust/ — 18 .rs files including per-dir mod.rs gateways + lib.rs

Each fixture exercises a different build_repo_graph.py language pattern (Python import, TS import from, Rust use) and the same hand-labeled task shape (e.g., "add rate limiting to login" → rate_limit.{py,ts,rs}, login.{py,ts,rs}, router).

Ground truth: 32 hand-labeled prompts in python/evals/autocontext_prompts.json (12 Python + 10 TS + 10 Rust), each mapped to the top files a competent engineer would open first.

Protocol: for each prompt, pipe {"prompt": ..., "cwd": fixture_root} into hooks/python/auto_context.py, parse the emitted <context-os:autocontext> block for ordered file paths (requiring / and a valid source extension), compare against expected_files.

Aggregate results (auto_context vs naive-filename baseline):

Per-fixture breakdown (auto_context): Python P@3 = 0.694, MRR = 1.000 · TypeScript P@3 = 0.767, MRR = 1.000 · Rust P@3 = 0.633, MRR = 0.950 · Coverage = 1.00 across all fixtures.

Baseline (--baseline naive-filename): ranks files by how many prompt tokens appear in the basename (camel-split + snake-split), tie-break shorter path. No graph, no imports, no hot-file boost. This is the trivial floor any static-RAG must beat; the +0.422 MRR lift is the graph doing real work.

Token-savings simulation (session_replay.py): a deterministic simulator that counts tokens "burned to first relevant file" in two flows on the 32-prompt corpus:

• without — 1 Glob (200 tok) + 1 Grep (500 tok) + Read files in baseline order × (line_count × 8 tok/line) until the first expected file is read.
• with — emit the auto_context block (measured tokens), Read files in predicted order; fall back to baseline flow if all top-K miss.

What MRR 0.922 / 80% savings means in practice: auto_context ranks the correct file first on 27/32 prompts; the 56-token cost of emitting the block is amortized within the first avoided Read on even a 10-line file. Across all three languages, the graph produces meaningful lift without embeddings or a server.

CI gate: autocontext eval exits non-zero if aggregate P@3 < 0.55 or MRR < 0.60; session_replay exits non-zero if win_rate < 50%. Both thresholds configurable via --threshold-* flags.

Where it falls short: per-fixture P@3 is pulled down by "hub" files (models.py with class Session/User/Token, mod.rs re-export files). Case-insensitive symbol match hits them for many prompts, bumping them into top-3 alongside the actually-targeted source file. These are genuinely useful neighbors but strict P@3 against hand-labeled ground truth penalizes them — MRR is the more operationally relevant metric since what matters is "did the user read the right file on the first Read".

Source: python/evals/reports/live-session-bench.md (summary) and python/evals/reports/live-session-bench-raw.json (per-call raw usage). Runner: python3 python/evals/runners/live_session_bench.py --runs 3. Requires the claude CLI + authenticated session.

Protocol: for each of 6 realistic developer prompts on autocontext_fixture/, invoke claude --print --output-format json --model sonnet --permission-mode bypassPermissions three times per arm, toggling the hook via CONTEXT_OS_AUTOCONTEXT=0 (control) vs =1 (treatment). Parse Claude Code's returned usage block; total_tokens = input + cache_creation + cache_read + output. Same fixture, same model, same cold cache per invocation — only variable is hook on/off.

Results (N=36 real Claude calls, $1.04 total cost):

What 40.9% means: this is the real number — tokens billed by the Anthropic API, with Claude actually running Read/Glob/Grep. The 80.2% simulation in §7.3 is an upper-bound estimate; the live bench is ground truth. The gap (80% simulated → 41% live) is because real Claude pays cold-cache cost on the system prompt + tool schemas every --print call, so the auto_context block's relative contribution is diluted. On long interactive sessions where the first-turn cost amortizes, we'd expect the savings to move up the range toward the simulation.

All six prompts win, including two "lookup only" prompts (p1, p2) where the single-run A/B in an earlier trial showed variance-driven regressions. With --runs 3 averaging, the signal stabilizes.

What's still not measured:

• Multi-turn interactive sessions (>20 turns).
• Large real repos (>10k files).
• Long-tail prompt diversity (6 prompts is a starting sample).

Source: python/evals/reports/benchmark-fixture-20260415.json, run 2026-04-15 (single run, N=1).

target:  /tmp/cos-bench-test  (2 files: README.md, one .js)
task:    "List the top-level directory structure of this repo,
          then count source files by language. One line per language."
model:   sonnet

What to read from this:

1. Most of the token delta is cache_read_input_tokens (−25,882). This is the system prompt / tool schema load. On a 2-file repo there is effectively no "user content" to filter — the saving comes from Claude Code loading a leaner working set because .claudeignore blocks the phantom directories the scanner would otherwise stat.
2. Output-token delta is −172 (−36.9%). Consistent with the upstream 40–65% claim for response shaping on explanatory prompts.
3. Cache-creation ticks up (+301). This is the Context OS CLAUDE.md + settings overhead. As predicted in §4.5.
4. Cost reduction (18.8%) is lower than token reduction (32.3%) because cache reads are cheap ($0.30/M). The composition of savings matters.
5. N=1. No variance. Do not generalize.

For each technique in the README, we tag the evidence supporting its token-savings claim. Four classes:

• M — directly measured in this repo, with a report artifact linked.
• U — cited from an upstream measurement we have read and consider credible.
• D — derived from documented Anthropic behavior (env vars, pricing, etc.) with arithmetic we can show.
• N — non-token (UX / safety / correctness). No token claim.
• P — pending measurement. We have a plan but no number yet.

Bottom line: of 18 techniques, 3 have direct measurements in this repo, 2 have upstream measurements, and 8 are arithmetic derivations from documented behavior. Auto-context (added v2.3.0) is the newest measured technique — MRR 0.958 on the hand-labeled fixture means the first injected candidate is correct for 11 of 12 prompts.

• Running the current N=1 benchmark on 50 more repos. Without N-within-cell variance, cross-repo comparisons still have no CI. Better to do 5 runs on 5 pinned fixtures than 1 run on 50 random repos.
• Switching to token-counting via tiktoken / local estimates. Claude's server-reported token counts are authoritative; any local approximation adds a confound.
• Claiming "up to X%" numbers. We do not quote maxima.

Nothing strictly required, but would help:

• Publish cache eviction policy (or statistical distribution of TTL-within-TTL).
• Expose a deterministic mode for benchmarking (temperature=0, seed=<n> behavior across sessions).
• Document the exact composition of system-prompt / tool-schema cache entries so we can attribute cache-read savings precisely.

• Cache read (cache_read_input_tokens) — tokens served from the prompt-cache tier. Cheap ($0.30/M on Sonnet).
• Cache creation / write (cache_creation_input_tokens) — tokens written into the cache this turn. Charged at a premium over fresh input ($3.75–$6/M depending on TTL).
• Fresh input (input_tokens) — tokens in the prompt that were neither cached nor cache-written. Charged at standard input rate ($3/M on Sonnet).
• Output (output_tokens) — model-generated tokens. Most expensive class ($15/M on Sonnet).
• Session — one claude invocation from start to stop, identified by session id.
• Task — one user prompt until the model returns stop_reason without a continuing tool call.
• Fixture — a pinned, reproducible input artifact (repo or log file) used in a benchmark.
• Protected string — a literal (file path, version, shell command, error class) that a safe-mode reducer must not drop.
• Ablation — running the full system with one technique disabled, to measure that technique's marginal contribution.

Everything referenced by this document, for cross-checking:

• README.md — headline claims.
• setup.sh — installer and --measure estimator.
• scripts/benchmark.sh — end-to-end benchmark runner.
• python/evals/runners/safe_mode_runner.py — reducer benchmark runner.
• python/evals/runners/compaction_survival_runner.py — session-memory correctness runner.
• python/evals/datasets/safe_mode_cases.json — reducer fixture manifest.
• python/evals/datasets/compaction_survival_cases.json — session-memory fixture manifest.
• python/evals/reports/safe-mode-report.{json,md} — latest reducer benchmark.
• python/evals/reports/compaction-survival-report.{json,md} — latest session-memory benchmark.
• python/evals/reports/benchmark-fixture-20260415.json — the single-run end-to-end result.
• docs/benchmark-plan.md — pre-existing benchmark scope notes (superseded by this document on the measurement-rigour dimension).
• docs/limitations.md — shipping-time honest limitations.

Methodological objections are the whole point of publishing this document. Open an issue at github.com/sravan27/context-os with the tag methodology and we will address it in the change log.