context-bench

Measure whether your LLM context system actually works — in one command.

You built something that sits between a user and an LLM: a context compressor, a RAG pipeline, a memory system, a reranker. Does it help? Does it hurt? Is it worth the tokens? context-bench runs your system against 42+ standard datasets so you don't have to build your own eval harness.

LoCoMo tests whether a system can answer questions about 400–700 turn conversations — temporal reasoning, multi-hop lookups, adversarial traps. The original paper's best baseline (GPT-3.5-turbo-16K) scores 37.8% F1. Human ceiling is 87.9% F1.

Full benchmark — all 10 conversations, all 1,986 questions, scored with both token-level F1 and an LLM judge (Claude Sonnet as judge):

Mem0 was run via uv sync --extra mem0 with local HuggingFace embeddings and the same Claude Sonnet backend as all other systems. Its extraction-based approach (LLM calls during ingest) loses significant detail from long conversations — the extract step takes ~90 min for one 419-turn conversation.

Per question type (RLM system):

How this compares to published systems:

The RLM system matches the original paper's best baseline on raw F1 using only open-source components (sentence-transformers, LanceDB, DuckDB). Mem0's published 66.9% LLM-Judge score uses their managed cloud platform with GPT-4 as judge — our local OSS run with Claude Sonnet as judge scored 27.6%.

Note on metrics: The original paper uses token-level F1. Most 2025+ systems report LLM-as-Judge scores, which are much more lenient (e.g. MemMachine reports 91.7% judge but only ~25% F1). Numbers across different metrics are not directly comparable. Judge model choice also matters — our results use Claude Sonnet, others use GPT-4/4o.

git clone https://github.com/npow/context-bench.git
cd context-bench
uv sync

# You need an OpenAI-compatible LLM endpoint (OpenAI, Anthropic relay, vLLM, Ollama, etc.)
# Full run (~7h for 3 systems x 10 conversations x 1,986 queries):
uv run python3 run_full_locomo.py

# Quick version (~15 min, 1 conversation):
context-bench memory \
  --system naive --system embedding --system rlm \
  --relay http://localhost:8080 \
  --dataset locomo -n 1

from context_bench.datasets.memory.locomo import locomo
from context_bench.systems.rlm import RLMSystem
from context_bench.systems.naive import NaiveSystem
from context_bench.systems.embedding import EmbeddingSystem
from context_bench.evaluators.answer_quality import AnswerQuality
from context_bench.evaluators.llm_judge_locomo import LLMJudgeLoCoMo
from context_bench.memory_runner import evaluate_memory

examples = locomo()  # all 10 conversations
systems = [
    NaiveSystem(base_url="http://localhost:8080", model="gpt-4", api_key="..."),
    EmbeddingSystem(base_url="http://localhost:8080", model="gpt-4", top_k=50, api_key="..."),
    RLMSystem(base_url="http://localhost:8080", model="gpt-4", api_key="..."),
]
evaluators = [AnswerQuality(), LLMJudgeLoCoMo(relay_url="http://localhost:8080", model="gpt-4")]

result = evaluate_memory(systems=systems, dataset=examples, evaluators=evaluators)

LongMemEval tests memory over ~5,000-turn conversations (~776K words each) — much longer than LoCoMo. Six question types: knowledge updates, multi-session recall, single-session facts, preferences, and temporal reasoning.

30 examples (5 per question type), Claude Sonnet as both answerer and judge:

Per question type (embedding system):

Context stuffing (naive) completely fails on 5,000-turn conversations. Embedding and RLM both handle it — embedding wins on this benchmark because pure semantic similarity works well for direct recall, while RLM's entity/keyword strategies add less value when conversations are this long and diverse.

uv run python3 run_full_longmemeval.py

Point it at any system that speaks /v1/chat/completions:

context-bench \
  --proxy http://localhost:7878 --name my-system \
  --proxy http://localhost:9091 --name baseline \
  --dataset hotpotqa -n 50

Evaluators are auto-wired — F1 for QA, code execution for HumanEval, LaTeX-aware matching for math, ROUGE for summarization. 42+ datasets across 10 categories.

The autoresearch loop uses Claude to iteratively mutate and test a retrieval pipeline:

uv run python3 loop.py \
  --relay http://localhost:8080 \
  --iterations 50 \
  --output-dir loop_results/ \
  --pipeline-path pipeline/sota_pipeline.py

Each iteration: Claude reads the pipeline source + score history, proposes one architectural change, evaluates it, keeps improvements. Run it overnight with the watchdog script.

git clone https://github.com/npow/context-bench.git
cd context-bench
uv sync                       # core deps
uv sync --extra datasets      # + HuggingFace dataset loaders

Requires Python 3.10+ and uv.

You need an OpenAI-compatible LLM endpoint for systems that call an LLM. Any endpoint that serves /v1/chat/completions works — OpenAI, Anthropic via a proxy, vLLM, Ollama, etc.

context-bench memory \
  --system naive --system embedding --system rlm \
  --relay http://localhost:8080 \
  --dataset locomo -n 1

context-bench \
  --proxy http://localhost:8080 \
  --dataset hotpotqa -n 20

from context_bench import evaluate, OpenAIProxy
from context_bench.evaluators import AnswerQuality
from context_bench.metrics import MeanScore, PassRate

result = evaluate(
    systems=[OpenAIProxy("http://localhost:8080", model="gpt-4")],
    dataset=your_data,
    evaluators=[AnswerQuality()],
    metrics=[MeanScore(score_field="f1"), PassRate(score_field="f1")],
)
print(result.summary)

Every evaluation follows four steps:

Dataset → System → Evaluator → Metric

1. Dataset — examples with questions, context, and ground-truth answers
2. System — your thing: compresses, retrieves, transforms, or just forwards
3. Evaluator — scores the output against ground truth
4. Metric — aggregates across examples (mean score, pass rate, cost per pass, latency)

All interfaces are typing.Protocol — implement the methods, don't subclass. A complete custom system is 5 lines:

class MySystem:
    name = "my-system"
    def process(self, example):
        transformed = my_transform(example["context"])
        return {**example, "context": transformed, "response": transformed}

For stateful memory systems:

class MyMemory:
    name = "my-memory"
    def reset(self): ...
    def ingest(self, items: list[Item]) -> IngestResult: ...
    def query(self, question: str, budget: int | None = None) -> QueryResult: ...

Pick a dataset name — the right evaluator is wired automatically.

Some are configurable: mmlu:anatomy, mgsm:de, bbh:causal_judgement, longbench:qasper.

Local JSONL files work too: --dataset ./my_data.jsonl

The memory benchmark tests whether a system can answer questions about conversations it ingested.

Item types the system receives during ingestion:

Evaluators:

Filter by QA type:

context-bench memory \
  --system rlm --relay http://localhost:8080 \
  --dataset locomo -n 3 --qa-types temporal,multi_hop

The loop (loop.py) evolves a retrieval pipeline by having Claude propose mutations.

Cycle: read pipeline source + scores → Claude proposes one change → evaluate → keep if better → repeat.

uv run python3 loop.py \
  --relay http://localhost:8080 \
  --model sonnet \
  --dataset locomo \
  --iterations 50 \
  --eval-n 2 \
  --seed 42 \
  --output-dir loop_results/ \
  --pipeline-path pipeline/sota_pipeline.py \
  --resume

Starting pipelines:

• sota_pipeline.py — coreference resolution, entity-relation triples, temporal ranking, query decomposition, multi-hop retrieval, embedding fallback
• entity_pipeline.py — simpler entity extraction baseline

Overnight operation:

bash watchdog6.sh   # checks every 10 min, restarts on crash/stall

Output:

loop_results/
├── loop_log.jsonl       # every iteration: score, mutation, accepted?
├── best_pipeline.py     # current best pipeline
├── baseline_score.json  # cached baseline
├── run.log              # full stdout
└── watchdog.log         # restart events

context-bench [options]           # benchmark context proxies
context-bench memory [options]    # benchmark memory systems

Proxy benchmark:

Memory benchmark:

uv sync                    # core
uv sync --extra datasets   # + HuggingFace dataset loaders
uv sync --extra mem0       # + Mem0 integration
uv sync --extra zep        # + Zep integration
uv sync --extra dspy       # + DSPy optimizer
uv sync --all-extras       # everything
uv sync --group dev        # development (pytest)

uv run pytest

src/context_bench/
├── __main__.py          # CLI (context-bench command)
├── types.py             # Protocol definitions (System, Evaluator, Metric)
├── memory_types.py      # Memory protocol + typed items
├── runner.py            # evaluate() orchestration
├── memory_runner.py     # evaluate_memory() orchestration
├── results.py           # EvalRow / EvalResult
├── cache.py             # JSONL caching for resume
├── systems/             # OpenAIProxy, RLM, Embedding, Mem0, Zep, Naive, ...
├── datasets/            # 42+ loaders (includes memory/locomo, memory/longmemeval)
├── evaluators/          # 11 evaluators
├── metrics/             # 7 metrics
├── reporters/           # Markdown, JSON, HTML output
├── loop/mutator.py      # LLM-driven pipeline mutator
└── dspy_bench/          # DSPy optimizer integration

pipeline/
├── sota_pipeline.py     # Strong baseline for autoresearch
└── entity_pipeline.py   # Simple entity-based baseline

MIT