BKP-500 — Bharat Knowledge Probe

Evaluation harness for BKP-500, a benchmark of whether language models know India-specific locale conventions.

BKP-500 tests knowledge of Indian numeral magnitudes and digit grouping (lakh/crore), state-specific land units (bigha, katha, guntha), traditional mass units, the Indian fiscal year, agricultural crop seasons, government schemes and structural identifiers (PAN, GSTIN, IFSC, PIN codes). Scoring is fully deterministic, with no LLM judge. This repo is the code that runs a model against the corpus and grades the responses: model adapters (OpenAI, Anthropic, HF transformers, local OpenAI-compatible servers), numeric, categorical and date graders, and scoring, plus bharat_units, the reference normalizer library the gold answers were computed against. No benchmark data ships here. Benchmark page: sthanika.ai/benchmarks/bkp-500

Requires Python 3.10 or newer. The corpus is auto-gated on Hugging Face: accept its terms on the dataset page, then log in once.

git clone https://github.com/sthanika-ai/Bharat-Knowledge-Probe-Benchmark.git
cd Bharat-Knowledge-Probe-Benchmark
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pip install -r requirements.txt    # torch, transformers, accelerate: needed by the hf backend

# Needs access granted on the dataset page first, otherwise the download returns 401
huggingface-cli login

# Corpus -> data/items/*.jsonl, the layout load_corpus() reads by default
python -c "from huggingface_hub import snapshot_download; snapshot_download('sthanika-ai/Bharat-Knowledge-Probe-Benchmark', repo_type='dataset', local_dir='.', allow_patterns=['data/items/*.jsonl'])"

# Confirm the install is sound
pytest

# The run script lives in BKP-500-model-runs, not here: copy it into this checkout
git clone https://github.com/sthanika-ai/BKP-500-model-runs.git ../BKP-500-model-runs
mkdir -p scripts && cp ../BKP-500-model-runs/configs/run_eval.py scripts/run_eval.py

# Quick test: 2 items on a model that needs no server (the model downloads from Hugging Face)
python scripts/run_eval.py --model gemma3-4b --mode smoke --limit 2 --device cuda:0 --batch-size 2
python scripts/run_eval.py --model gemma3-4b --mode full  --limit 2 --device cuda:0 --batch-size 2
python -m bkp_eval.score results/gemma3-4b/full.jsonl

# Real run (3 samples, both regimes), then score it
python scripts/run_eval.py --model <name> --mode full
python -m bkp_eval.score results/<name>/full.jsonl

Notes:

• The quick test populates only one category with 2 items, so n=0 for the other categories in the score report is expected.
• Most hf entries in MODELS default to --device cuda:1. On a single GPU, pass --device cuda:0 as above.
• vllm and ollama entries need a server you start yourself, because run_eval.py does not launch one. For vLLM: vllm serve <hf_repo> --port <vllm_port> --served-model-name <served_model_name>, using the values from that model's MODELS entry.
• --model must be a key in MODELS inside run_eval.py. This repo ships no run script: the exact script and per-model configs for the 19 published models are in BKP-500-model-runs. MODELS there is a registry you can extend: add your own entry for a model that isn't listed.
• Self-hosted models (vLLM, Ollama) need no API key. The harness talks to them over a local OpenAI-compatible endpoint via LocalAPIAdapter. For OpenAI or Anthropic models, pip install openai or pip install anthropic and export OPENAI_API_KEY or ANTHROPIC_API_KEY. The cloud adapters are implemented but have not been run against a live key.
• To load the corpus from the Hub in Python instead: pip install -e ".[hf-datasets]", then from bkp_eval.items import load_corpus_from_hf.
• python -m bkp_eval.score reports Bharat Score, per-category accuracy, unit discipline, overconfidence, refusal rate and consistency across repeated samples.
• bharat_units' CLI is a stub; use the library modules directly (bharat_units.numerals, .mass, .fiscal, .seasons, .identifiers).
• Lint and type checks: ruff check . and mypy src.

Full report: sthanika.ai/research/bkp500-2026 · Model runs: BKP-500-model-runs. Ranked by Bharat Score (macro-mean accuracy across all 7 categories), tiered by chained bootstrap-CI overlap.

OOM error rate is the share of numeric answers off by 10× or more (a lakh/crore-scale slip). Unit discipline is the share of numeric_with_unit answers that included an explicit, correct unit. CIs use n_resamples=2,000 (the harness default for a publication-grade run is 10,000).

Per-category accuracy (%), all 19 models × 7 categories:

Qwen3.6 27B, a global open-weight model, leads the roster. Sarvam-M 24B is the strongest India-built model at rank 3. Land units by state is the weakest category for 14 of the 19 models.

Exact run configuration for every model: BKP-500-model-runs · Report page: sthanika.ai

Cite the benchmark and this harness as in CITATION.cff. To cite the leaderboard and run configurations, cite the BKP-500-model-runs repository.

@misc{bkp500,
  title  = {Bharat Knowledge Probe (BKP-500): Does Your Model Know Where It Is?},
  author = {{Sthānika AI}},
  year   = {2026},
  url    = {https://github.com/sthanika-ai/Bharat-Knowledge-Probe-Benchmark}
}

Apache-2.0 for the harness code, see LICENSE. The dataset is CC-BY-NC-4.0; see the dataset card.

• BKP-500 dataset: the corpus, on Hugging Face. Corrections to the data itself belong there.
• BKP-500-model-runs: run configuration and leaderboard for every evaluated model
• Site: sthanika.ai