llm-extraction-evals

An evaluation harness for document field extraction — the tooling behind two questions every AI document project must answer before go-live:

1. Is the LLM actually better than the simple baseline? Per-field precision / recall / F1 against a labeled set, with an honest regex baseline included. If the model can't beat the baseline, ship the baseline.
2. Where should the confidence threshold sit? A sweep showing the automation-rate vs precision trade-off — the number that decides how much work lands on the human review queue vs flows straight through.

Runs fully offline (stdlib only). The Anthropic LLM extractor is an optional extra.

threshold ↑  →  precision ↑  automation ↓  (more goes to humans)
threshold ↓  →  precision ↓  automation ↑  (more errors reach analytics)

An abstention on a present field is a false negative — a human fills it in, costing minutes. A confident wrong answer is a false positive — it flows into reports, costing correctness. These are not symmetric costs, which is why the threshold is a business decision and deserves a table, not a gut call:

(illustrative — run the harness for real numbers)

git clone https://github.com/reshma449/llm-extraction-evals.git
cd llm-extraction-evals
pip install -e ".[dev]"

pytest -v                                 # run the test suite
python -m extraction_evals                # evaluate the baseline on bundled data
cat eval_report.md

# with an LLM extractor (optional)
pip install -e ".[llm]"
export ANTHROPIC_API_KEY=...
python -m extraction_evals --extractor anthropic

JSONL, one labeled document per line:

{"doc_id": "d01", "text": "Invoice Number: INV-20441 ...", "labels": {"invoice_number": "INV-20441", "po_number": null}}

A null label means the field is truly absent — predicting anything for it counts against the extractor. That's what keeps precision honest on documents where fields legitimately don't exist (10–30% of real corpora, in my experience).

• Strict matching, light normalization. Case, whitespace, and currency commas are normalized; everything else must match exactly. Fuzzy matching hides exactly the errors that burn you in finance data.
• The baseline is a first-class citizen, not a strawman: anchored patterns with confidence scores, the same interface as the LLM extractor (predict(text) -> {field: (value, confidence)}). New extractors implement one method.
• Deterministic and offline by default so CI runs the full suite on every push with no API keys or flakes.

src/extraction_evals/
  dataset.py     # JSONL loading + validation
  extractors.py  # BaselineRegexExtractor, AnthropicExtractor (optional)
  metrics.py     # precision/recall/F1, normalization, threshold sweep
  report.py      # Markdown report rendering
  __main__.py    # CLI
data/labeled.jsonl  # 10-document labeled sample set
tests/

MIT