HALLMARK

HALLucination benchMARK: A benchmark for evaluating citation hallucination detection tools.

The NeurIPS 2025 incident---where 53 papers were found to contain fabricated citations that passed peer review---exposed a critical gap: we have no standardized way to measure how well tools detect citation hallucinations. HALLMARK fills this gap.

HALLMARK draws on best practices from established benchmarks:

• HumanEval: Multi-criteria sub-tests per entry (~6 checks per citation)
• SWE-bench: Contamination awareness via temporal segmentation
• LiveCodeBench: Continuous updates and post-cutoff evaluation
• ONEBench: Sample-level atomic evaluation with ever-expanding pool

• Hallucination taxonomy: 14 types across 3 difficulty tiers (Easy / Medium / Hard)
• 2,526 annotated entries: 826 valid + 1,246 hallucinated with ground truth across the public splits, plus a 454-entry hidden split
• 6 sub-tests per entry: DOI resolution, title matching, author consistency, venue verification, field completeness, cross-database agreement
• Evaluation metrics: Detection Rate, F1, tier-weighted F1, detect@k (fraction detected within k strategies), ECE (expected calibration error)
• Built-in baselines: DOI-only, bibtex-updater, HaRC, verify-citations, LLM-based (OpenAI, Anthropic, OpenRouter), agentic LLMs with tool use, ensemble, DB-first cascade with hallucination-mode diagnosis, plus ports of two recent papers — hallucitechecker (Sakai et al. 2026) and checkifexist (Abbonato 2026 Algorithm 1) (CiteVerifier and hallucinator are available as wrapper modules but not registered in the default registry)
• Baseline registry: Central discovery, availability checking, and dispatch for all baselines (19+ variants)
• Reproducible runs: opt-in --cache-path flag wraps HTTP calls in a SQLite-backed requests-cache so re-runs reuse frozen API responses; --timing-breakdown and --subtask-diagnostic surface per-baseline performance + recognition/matching/calibration decomposition
• Plackett-Luce ranking: ONEBench-inspired ranking that handles incomplete evaluation data
• Automated execution: Orchestrator script and CI workflow for batch baseline evaluation
• Temporal analysis: Contamination detection via pre/post-cutoff comparison
• Community contributions: ONEBench-style ever-expanding sample pool

cascade_db_diagnosis — Stage 1 bibtex-updater + Stage 2 Claude Sonnet 4.6 (via OpenRouter, up to 5 tool calls), conservative vs aggressive scoring of residual UNCERTAIN:

Every released retrieval-dependent run predates EvaluationResult.source_condition and records no incomplete-lookup fraction; re-runs will carry it.

Aggressive promotion of residual UNCERTAIN (the "DB-as-gold-standard" stance) lifts Tier-3 F1 by +2.05 pp on dev_public / +1.07 pp on test_public at +4.00 / +4.84 pp FPR; the trade is paid in rank-discrimination (AUROC −1.34 / −1.47 pp). Full JSONs (incl. per-tier/per-type breakdowns) in data/v1.2/baseline_results/; see paper §Stage-2 diagnosis cascade for analysis.

# Recommended: clone and install in development mode
git clone https://github.com/rpatrik96/hallmark.git
cd hallmark
uv pip install -e ".[dev]"

# With LLM baseline SDKs (openai, anthropic)
uv pip install -e ".[baselines]"

# With ranking support (Plackett-Luce model via choix)
uv pip install -e ".[ranking]"

# All optional dependencies
uv pip install -e ".[all]"

Note: pip install hallmark is not yet published to PyPI. Use the clone + install path above.

Set HALLMARK_BIBTEX_CHECK_BIN to pin the bibtex-check executable and HALLMARK_BIBTEX_CHECK_RATE_LIMIT to set its per-service request rate. HALLMARK_BTU_ABSTENTION_AS_VALID=1 restores the legacy abstention-to-VALID mapping and changes the reported FPR. S2_API_KEY authenticates Semantic Scholar, while BIBTEX_CHECK_MAILTO selects CrossRef's polite pool. HALLMARK_ALLOW_SOURCE_OUTAGE=1 overrides the outage guard and produces numbers that are not comparable to a healthy run.

bibtex-check exits 5 when its sources go dark, the wrapper raises SourceOutageError, and HALLMARK refuses to score that run. Before any multi-hour run, use python scripts/check_source_reachability.py --require dblp,openalex: exit 0 passes, exit 1 means a required source is down, and exit 2 means nothing was probed.

The [baselines] extra installs only the LLM SDKs (openai, anthropic). External CLI tools require separate installation due to a bibtexparser 1.x dependency conflict:

# HaRC
pipx install harcx

# bibtex-updater. The wrapper targets >=1.11.0, which separates fields the
# checker refuted (`mismatched_fields`) from fields it declined to compare
# (`unconfirmed_fields`); older releases still parse, minus that distinction.
pipx install "bibtex-updater>=1.11.0"
# To reproduce released numbers, pin the release that produced them:
#   1.2.0 (scripts/regen_btu_v1_2_0.py, 2026-05-31) produced
#   data/v1.2/baseline_results/bibtexupdater_{dev,test}_public.json:
#   dev DR .865 / FPR .092, test DR .877 / FPR .115
#   Walters & Wilder supplement -> 1.4.0  (docs/walters_wilder_supplement.md, 2026-07)

# verify-citations
pipx install verify-citations

# CiteVerifier (GhostCite) — clone required
git clone https://github.com/NKU-AOSP-Lab/CiteVerifier

# hallucinator — clone required
git clone https://github.com/gianlucasb/hallucinator

Using pipx isolates each tool's bibtexparser 1.x from your project environment.

# Run DOI-only baseline on the dev split
hallmark evaluate --split dev_public --baseline doi_only

# Run the v1.1 cascade with aggressive scoring (DB as gold standard).
# Stage 2 LLM diagnoser is routed through OpenRouter — set OPENROUTER_API_KEY.
hallmark evaluate --split dev_public --baseline cascade_db_diagnosis_aggressive \
    --stage2-baseline llm_agentic_openrouter_claude_sonnet_4_6

# Re-score the same predictions under both eval modes (conservative + aggressive)
# in a single payload — the gap quantifies the abstention/indexing-lag tax.
hallmark evaluate --split dev_public --baseline cascade_db_diagnosis \
    --stage2-baseline llm_agentic_openrouter_claude_sonnet_4_6 \
    --eval-mode both

# Run with custom predictions
hallmark evaluate --split dev_public --predictions my_predictions.jsonl --tool-name my-tool

hallmark stats --split dev_public

# Run all free baselines and generate leaderboard
python scripts/run_all_baselines.py --split dev_public --output-dir results/

# Run specific baselines in parallel
python scripts/run_all_baselines.py --baselines doi_only,bibtexupdater --parallel

# Run only free (no API key) baselines, skip unavailable
python scripts/run_all_baselines.py --baselines free --skip-unavailable

For LLM-based baselines that take >1 hour sequentially, use the parallel-resume scripts to checkpoint and resume:

# Resume zero-shot LLM baselines (--provider openrouter|huggingface)
python scripts/parallel_resume_test_public.py \
    --checkpoint-dir results/checkpoints/llm_openrouter_deepseek_r1_test_public \
    --model deepseek/deepseek-r1 \
    --jsonl-name openrouter_deepseek_deepseek-r1.jsonl \
    --workers 4

# Qwen3 dense sweep runs on the HuggingFace router (needs HF_TOKEN)
python scripts/parallel_resume_test_public.py \
    --provider huggingface \
    --checkpoint-dir results/checkpoints/llm_hf_qwen3_4b_test_public \
    --model Qwen/Qwen3-4B:featherless-ai \
    --jsonl-name huggingface_qwen3-4b.jsonl \
    --workers 4

# Resume agentic verifiers (BTU, multi-tool, tool-augmented) with Sonnet 4.6
python scripts/parallel_agentic_btu_test_public.py \
    --checkpoint-dir results/checkpoints/agentic_btu_openai_test_public \
    --split test_public --verifier agentic_btu_openai

Both scripts support checkpointing and can safely resume interrupted runs without recomputing completed entries. Add --dry-run for a safe first invocation.

hallmark leaderboard --results-dir results/

See examples/ for full walkthroughs, including writing a custom baseline and per-type analysis.

To evaluate any external tool against HALLMARK, produce a JSONL file with one prediction per line and run:

hallmark evaluate --predictions my_preds.jsonl --split dev_public

Each prediction must include:

{
  "bibtex_key": "a3f9c2b1...",
  "label": "HALLUCINATED",
  "confidence": 0.87,
  "reason": "DOI does not resolve",
  "subtest_results": {"doi_resolves": false},
  "api_sources_queried": ["crossref"],
  "wall_clock_seconds": 1.2,
  "api_calls": 1
}

bibtex_key format: Keys in the benchmark are hex hashes (e.g., a3f9c2b1d4e7...), not human-readable keys like vaswani2017attention. Your predictions must use the exact keys from the loaded entries — use entry.bibtex_key when iterating over load_split() results.

See examples/03_custom_baseline.py for a complete end-to-end example.

UNCERTAIN label: UNCERTAIN is accepted as a prediction label. UNCERTAIN predictions are treated as VALID for confusion-matrix metrics (conservative default) and excluded from AUROC/AUPRC. Prefer VALID or HALLUCINATED with calibrated confidence when possible.

Confidence semantics: confidence = P(your predicted label is correct). If you predict HALLUCINATED with 0.9, you claim 90% certainty it is hallucinated. If you predict VALID with 0.8, you claim 80% certainty it is valid. This is NOT P(HALLUCINATED).

api_calls on the released bibtex-updater rows: bibtex-check wrote the sources that returned a matching record, so for those rows the reported mean_api_calls counts hits rather than calls. Runs with bibtex-updater 1.12 or later record the queried sources and count calls.

wrong_venue, preprint_as_published, partial_author_list and arxiv_version_mismatch are conditions on real works. An entry carrying one of them cites a paper that exists and says something inaccurate about it, usually a stale venue string or a preprint cited by its conference version. They sit under a label the agentic system prompt defines as "a HALLUCINATED (fabricated) citation", so a tool that correctly identifies a stale venue is scored as having found a fabrication. They are 26.1% of the positive class on dev_public and 26.8% on test_public.

The NeurIPS 2026 program chairs draw the same line for their area chairs. Their criteria, adopted from the ICLR and ICML 2026 guidance and sent to authors on 5 September 2026, count a reference as hallucinated when no document with a title anywhere near the one given can be found, when the author list adds non-authors or drops a large fraction of them, or when there is no evidence the venue exists; a real venue that is wrong for the reference, including a wrong journal name or arXiv ID, and small title or author errors are errors the authors are told to fix, not hallucinations. Under those criteria wrong_venue, preprint_as_published and arxiv_version_mismatch fall outside the label, near_miss_title and partial_author_list fall outside unless the error is large, and fabricated_doi and future_date are not covered at all. The criteria carry no public URL; the source is the program chairs' message to authors of 5 September 2026.

The ranking is not stable if they stop counting as fabrication. Scoring every tool three ways — as shipped, with the four modes out of the scored set, and with them as negatives so flagging one is a false accusation:

The last column folds out the seven modes the NeurIPS 2026 criteria leave outside the label rather than the four this section is about, from python scripts/ablate_taxonomy_fold.py --split test_public --fold neurips_2026.

The top of the table changes identity: cascade_db_diagnosis leads as shipped at MCC 0.897 on test_public, and bibtexupdater takes first place under as-false-positives on both splits (MCC 0.685 against the cascade's 0.631 on test_public, 0.733 against 0.678 on dev_public).

How much of a tool's credit comes from these modes varies widely, and no published column shows it. On test_public they are 27.0% of the cascade's 514 detections, 23.2% of Sonnet 4.6's, 20.0% of bibtexupdater's and 9.1% of doi_only's. Detection rate on the four ranges from 0.065 (doi_only) to 1.000 (the cascade), median 0.740 across 20 tools. The dev_public count excludes the Opus 4.7 result, whose per-type block predates the relabel and does not reproduce its published MCC; it is registered known-stale in scripts/check_results_freshness.py.

Two controls bound what this means. Folding hybrid_fabrication alone leaves the folded-out ranking unchanged on both public splits — tau 1.000, no tool changing position — and moves six tools under as-false-positives (tau 0.968 and 0.965), so the instability belongs to the four modes rather than to folding a mode class, though at 4.3% and 5.6% of positives that null carries less weight than the effect it contrasts with.

Surface features alone — field presence, string lengths, author count, entry type, year, no lookups — separate the four modes from VALID entries at DR 0.468 on dev_public and 0.388 on test_public against a majority baseline of 0.000, and conditioning the negatives the same way the positives are conditioned does not move it. That is real leakage and a benchmark limitation. It is also well short of the cascade's 1.000, so the cascade's score is not only the generator being read back.

Reproduce: uv run --with scikit-learn python scripts/shortcut_control_four_modes.py

stress_test inherits the same property by construction. It holds three modes, two of which are on this list: 75 of its 121 scored entries, 62%. Its one VALID entry is a contamination canary excluded from scoring, so its false-positive rate is undefined and its MCC is 0.0 by degeneracy rather than by measurement, and every one of the cascade's misses there is on the two real-paper modes — folding them takes it from 0.953 to 1.000.

No labels change in this release. Separating the two questions means relabelling across every split and moving every per-type and per-tier number, so the instability is reported rather than repaired; treat a ranking on the current target as a ranking over both questions at once. Reproduce with python scripts/ablate_taxonomy_fold.py --split test_public, which writes tables/taxonomy_fold_ablation_*.csv and rebuilds each tool's confusion matrix from per_type_metrics, validating it against that result's own published figures first. Discussion is in issue #36.

The dataset ships in this repository under data/v1.2/ as JSONL, so every split is available without an external download. The HuggingFace mirror published for the NeurIPS 2026 submission was withdrawn together with the submission.

A Croissant 1.0 metadata file is included at the repo root (croissant.json) and points at the files in this repository. It covers all public splits and includes RAI fields required by NeurIPS 2026 D&B. Validate locally with:

mlcroissant validate --jsonld croissant.json

stress_test design note: The stress_test split is all-hallucinated by design. It contains challenging edge cases (merged citations, partial author lists, arXiv version mismatches) intended to stress-test detection robustness beyond the main splits. Because there are no valid entries, FPR and specificity are undefined for this split. Use detection rate as the primary metric when reporting stress_test results.

Tier distribution per split: ~27% Tier 1, ~47% Tier 2, ~26% Tier 3 (hallucinated entries).

Each entry is a JSON object in JSONL format:

bibtex_key format: Keys are hex hashes (e.g., a3f9c2b1d4e7...), not human-readable keys. When writing predictions, always use entry.bibtex_key directly — do not construct keys manually.

{
  "bibtex_key": "a3f9c2b1d4e76f85",
  "bibtex_type": "inproceedings",
  "fields": {
    "title": "Attention Is All You Need",
    "author": "Ashish Vaswani and Noam Shazeer and ...",
    "year": "2017",
    "booktitle": "NeurIPS",
    "doi": "10.5555/3295222.3295349"
  },
  "label": "VALID",
  "hallucination_type": null,
  "difficulty_tier": null,
  "explanation": "Valid entry scraped from DBLP and verified",
  "subtests": {
    "doi_resolves": true,
    "title_exists": true,
    "authors_match": true,
    "venue_correct": true,
    "fields_complete": true,
    "cross_db_agreement": true
  }
}

The title_oracle baseline quantifies the ceiling of a perturbation-structure shortcut present in HALLMARK's design. Because most HALLUCINATED entries are generated by perturbing real (VALID) papers, they inherit the original title. This means a title that appears as VALID in the dev split almost certainly belongs to a perturbed — hence hallucinated — entry when it reappears in another split.

The oracle exploits this directly: if a blind entry's title matches any VALID title in the dev split, it predicts HALLUCINATED.

Empirical results on v1.0 data:

• ~33% of unique titles appear as both VALID and HALLUCINATED across dev/test splits.
• Applied to the hidden split: F1 = 0.389 at perfect precision (P = 1.0, recall = ~0.24).
• Titles absent from any valid pool are 100% HALLUCINATED in the dataset.

This is not a legitimate detection method — it requires access to dev ground-truth labels as a look-up table, which constitutes label leakage when evaluating on dev itself. Report it alongside real baselines to make the shortcut visible. Any real tool that achieves F1 below the title oracle on the hidden split is arguably exploiting benchmark structure rather than performing genuine citation verification.

from hallmark.baselines.title_oracle import run_title_oracle
from hallmark.dataset.loader import load_split

dev_entries = load_split("dev_public")
test_entries = load_split("test_public")
blind_test = [e.to_blind() for e in test_entries]

predictions = run_title_oracle(blind_test, reference_pool=dev_entries)

Tools evaluated on dev_public. All numbers reproduce Table 1 of the paper. Bold = best among independent zero-shot LLMs. ΔFPR is the cross-split shift test_public − dev_public; — means no test_public evaluation.

Coverage and relabel caveat. A tool that returns UNCERTAIN is excluded from the confusion matrix, so metrics use the entries it answered; coverage reports 1 - num_uncertain / num_entries. The Coverage column is dev_public only, so ΔFPR can compare two denominators: DeepSeek-R1 answered 651 of 831 entries on test_public against 1,101 of 1,119 on dev_public, and its −.303 is that gap rather than a robustness result. Every row moved under the current labels. The largest separate corrections are DOI-only FPR .195 → .043 after the HTTP-202 re-run (829d2d4), and bibtex-updater FPR .179 → .092 with its stored coverage field moving 1.000 → .862 (467da9d/3aff25b). See notes/eval-hardening-2026-09-04.md for the outcomes.

† Registered in KNOWN_STALE: the Opus 4.7 and Sonnet 4.6 dev files retain pre-relabel per-type blocks and need re-runs under the current labels.

Every released retrieval-dependent run predates EvaluationResult.source_condition and records no incomplete-lookup fraction; re-runs will carry it.

DR = Detection Rate · FPR = False Positive Rate · TW-F1 = Tier-weighted F1 · MCC = Matthews Correlation Coefficient · ECE = Expected Calibration Error. The canonical data for this table are in tables/main_results_dev_public.csv. The shaded co-designed block is a reference upper bound: bibtex-updater's development overlapped with the benchmark's taxonomy design, so its scores risk construct-overfitting and should not be compared head-to-head with independent tools. HaRC and verify-citations are omitted. For HaRC the stated reason — Semantic Scholar throttling collapsing effective coverage below 7% — does not survive inspection: given a key, HaRC runs at full coverage on dev_public (harc_with_s2key_dev_public.json). That keyed run is itself scored against the pre-relabel ground truth, so its detection rate and FPR are not comparable to the current-label results above and are not quoted here; it is registered as known-stale in scripts/check_results_freshness.py. Regenerating it needs a working Semantic Scholar key, which as of 2026-09-04 returns HTTP 403. The unkeyed harc_dev_public.json was withdrawn the same day: it scored 521 of 1,119 entries, matching no split, and reported FPR 0.000 — a truncated run, not a result.

Two caveats on that omission, both open. bibtexupdater.py reads S2_API_KEY from the environment automatically and the HaRC wrapper did not until 2026-09-04, so a default side-by-side run handed the co-designed tool authenticated access and its competitor the unauthenticated pool. And if HaRC leans on Semantic Scholar where bibtex-updater's cascade can fall through to CrossRef and OpenAlex, then "HaRC throttles" is partly a measurement of which tool depended on the source that was down rather than a property of the tools. That is not currently checkable: the HaRC wrapper records mean_api_calls: 0.0 and no per-source counts, so the released artifacts cannot answer it. The bibtex-updater side of that comparison is a hit count rather than a call count, for the reason given under Prediction Fields.

1. LLMs span a wide recall–precision spectrum. From ultra-conservative (Gemini 2.5 Pro: 48% DR, 5% FPR) to aggressive (DeepSeek-V3.2: 91% DR, 70% FPR). Sonnet 4.6 and Opus 4.7 tie and lead the independent tools on dev_public (Sonnet F1 = 0.827 / ECE = 0.066); on test_public, Sonnet leads in F1 (p = 0.0486). Both are far ahead of GPT-5.1 (F1 0.766) and the recall-aggressive open-weight cohort.

2. Agentic lookups inflate FPR. A 5-call budget overtakes bibtex-updater by 11.5 pp DR (0.98 vs. 0.86), but agentic FPR remains ~5× higher (0.47 vs. 0.09) because the harness flags an entry whenever any one of CrossRef/OpenAlex/arXiv returns no match. F1 still trails by 6.6 pp. Substituting Sonnet 4.6 reproduces the GPT-5.1 profile within ≤4.7 pp on every metric — the FPR rise is harness-driven, not LLM-driven.

3. Base-rate precision collapse. At a 1% base rate the twenty evaluated settings return between one true finding per ten flags (Opus 4.7) and one per eighty (DeepSeek-V3.2), and the cascade one per seventeen; at the 0.1% end the best is one per eighty-nine. Recall-optimized verifiers misallocate reviewer effort (see "What these numbers mean on a real bibliography").

4. Post-cutoff calibration breakdown. On 448 papers from 2024–2025, 8 of 12 LLMs over-flag sharply (FPR up to 0.89). Sonnet 4.6 and Opus 4.7 hold FPR ≤ 0.12; GPT-5.4 (FPR 0.41) and Gemini 2.5 Pro (FPR 0.25) only partially recover.

5. A capability gap remains. Even the highest-recall independent model misses 9% of hallucinations, with systematic weaknesses on subtle types (near_miss_title: 58%, author_mismatch: 45% for GPT-5.1). No tool dominates across regimes: bibtex-updater is cheapest and most temporally stable; Sonnet 4.6 / Opus 4.7 lead on FPR and PPV; the rule-based F1 lead collapses on test_public.

See the paper for the full per-tier, per-type, and temporal-robustness analyses, or explore them interactively on the companion website.

HALLMARK also wraps several external citation verification tools as baselines:

# Use the baseline registry to discover and run any baseline
from hallmark.baselines.registry import list_baselines, check_available, run_baseline
from hallmark.dataset.loader import load_split

entries = load_split("dev_public")

# List all registered baselines (or just the free ones)
print(list_baselines(free_only=True))

# Check if a baseline's dependencies are installed
available, msg = check_available("harc")

# Run a baseline by name
predictions = run_baseline("harc", entries)

See also:

• GhostCite paper — large-scale analysis of 2.2M citations across 56K papers
• HalluCitation paper — analysis of ~300 hallucinated papers in ACL conferences
• GPTZero Hallucination Detector — commercial API for citation verification

test_public is 62.5% hallucinated by construction, and that is the operating point every number above is measured at. Detection rate and false-positive rate are properties of a detector and do not move with prevalence, but precision does — and precision is what a user actually experiences, because it is the share of the flags handed to them that are real. A tool can therefore post an honest 99% detection rate at a 16% false-positive rate and still hand a user mostly false accusations.

Holding each tool's measured DR and FPR fixed and sweeping the assumed prevalence (python scripts/compute_base_rate_precision.py, full table in tables/base_rate_precision.csv):

The benchmark's own prevalence is the only point on that curve where the instrument looks good, and no arithmetic here is new — it is Bayes' rule over numbers already in data/v1.2/baseline_results/.

We ran HALLMARK's own cascade_db_diagnosis with an agentic Stage 2 over 5,043 deduplicated references from 267 real NeurIPS-workshop submissions. Stage 1 located a matching record for every reference it ruled on, and not one fabricated work was found in 5,043 references. Of those, 3,244 were confirmed outright, 920 differed from the indexed record only in the venue string, 187 carried some other metadata difference, 431 were flagged and 261 abstained.

Eleven accusations survived re-adjudication and hand audit, and none was a citation to something that does not exist:

So on real bibliographies the instrument's main output is false positives, and the flag list is dominated by citation-style artefacts and tool bugs rather than fabrication. That is worth stating plainly rather than leaving for a reader to derive: at a wild prevalence indistinguishable from zero, the left-hand column of the table above is the column that applies.

Two consequences for how these baselines should be read. Report precision at the prevalence you expect, not the benchmark's, and prefer a tool that abstains over one that guesses — a citation flagged as fabricated is a serious accusation against a named author, and at these base rates most such flags are wrong. The gap between "this work does not exist" and "this entry describes a real work incorrectly" carries most of the real signal, and the current taxonomy folds the second into the first (see issue #36); on the wild corpus, of the 324 references Stage 2 upheld as flags, 59.0% resolved to a real work described wrongly.

This analysis is joint work with the InterpScience submission-screening effort, which contributed the 5,043-reference corpus and its adjudications.

from hallmark.dataset.loader import load_split
from hallmark.evaluation.metrics import evaluate
from hallmark.dataset.schema import Prediction

# Load benchmark entries
entries = load_split("dev_public")

# Create predictions (your tool's output)
predictions = [Prediction(bibtex_key=e.bibtex_key, label="VALID", confidence=0.5) for e in entries]

# Evaluate
result = evaluate(entries, predictions, tool_name="my-tool", split_name="dev_public")
print(f"F1: {result.f1_hallucination:.3f}")
print(f"Detection Rate: {result.detection_rate:.3f}")

HALLMARK includes an ONEBench-inspired ranking system based on the Plackett-Luce model that handles incomplete evaluation data (not all tools evaluated on all entries):

from hallmark.evaluation.ranking import rank_tools_plackett_luce, rank_tools_mean_score

# Rank tools using Plackett-Luce (requires choix: pip install hallmark[ranking])
pl_ranking = rank_tools_plackett_luce(entry_keys, tool_names, matrix)

# Fallback: simple mean-score ranking (no extra dependencies)
mean_ranking = rank_tools_mean_score(entry_keys, tool_names, matrix)

HALLMARK includes two GitHub Actions workflows:

• tests.yml: Runs the full test suite across Python 3.10-3.13 on every push/PR
• baselines.yml: Runs live free baselines (doi_only, verify_citations) weekly and on demand; harc and bibtexupdater use pre-computed result validation (checksum checks) instead of live re-execution due to API rate limiting

HALLMARK uses an ever-expanding pool inspired by ONEBench. To contribute new entries:

hallmark contribute --file my_entries.jsonl --contributor "Your Name"

See CONTRIBUTING.md for details on entry format, validation requirements, and the review process.

hallmark/
├── hallmark/                  # Python package
│   ├── dataset/               # Schema, loader, scraper, generator
│   ├── evaluation/            # Metrics, subtests, aggregator, temporal, ranking
│   ├── baselines/             # Registry + baselines (DOI-only, bibtex-updater, LLM×6, ensemble, HaRC, CiteVerifier, hallucinator, verify-citations)
│   │   └── registry.py        # Central baseline discovery, availability, dispatch
│   ├── contribution/          # Pool manager, entry validation
│   └── cli.py                 # Command-line interface
├── data/
│   ├── v1.2/                  # Benchmark splits (dev_public, test_public, stress_test)
│   ├── hidden/                # Hidden test set (not public)
│   └── raw/                   # Raw scraped/generated entries
├── scripts/
│   └── run_all_baselines.py   # Batch orchestrator for baseline evaluation
├── .github/workflows/
│   ├── tests.yml              # CI: test suite across Python versions
│   └── baselines.yml          # CI: weekly free baseline evaluation
├── tests/                     # Test suite
├── figures/                   # Evaluation figures
└── examples/                  # Usage examples

If you use HALLMARK in your research, please cite:

@misc{reizinger2026hallmarkdiagnosingfailuremodes,
      title={HALLMARK: Diagnosing Three Failure Modes in LLM Citation Verifiers},
      author={Patrik Reizinger and Wieland Brendel},
      year={2026},
      eprint={2607.18360},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2607.18360},
}

MIT