laya-cuda-bench

How many typed decisions per second can one NVIDIA GPU serve under a p99 latency SLO, and what does each million cost? An open, reproducible capacity-planning study of Laya, Convai's open-weights "System 1" decision models (421M English / 322M multilingual), across four GPUs, five inference backends, MIG slicing, a hosted API and two LLM baselines.

Full study, with every table traced to the file that backs it: REPORT.md.

¹ Sweep predates the harness's downward search; listed as a follow-up in REPORT.md §11. Cost = card amortised over 3 years at 100 % utilisation + energy at $0.12/kWh (assumptions in §6.9).

Every figure is drawn from the committed results/ by scripts/make_figures.py, which asserts the headline numbers against this README and the report before drawing.

Past capacity, p99 goes from ~100 ms to seconds within one ×1.4 load step, because the batcher fills to max_batch and a batch of long notices becomes a ~1 s forward. Size below the knee, not at "GPU 100 %".

The two busiest hours (187 and 185 decisions/s) reach 132 and 143 ms. Holding 130 ms at peak needs ~25 % headroom or a faster backend.

torch.compile(mode="max-autotune") FP16 is the fastest static backend on every card (1.3–1.7× eager FP16 at batch ≥ 64). Under dynamic-batch serving the winner depends on architecture: TensorRT on Hopper (H100: 175 vs 93 decisions/s at p99 ≤ 130 ms), eager FP16 ties TensorRT on Blackwell (RTX PRO 6000: 146 vs 146).

flowchart LR
    W["1,000 public SAM.gov notices<br/>3-question bundle per event"] --> P{"Parity gate<br/>63/63 vs upstream FP32"}
    P -->|pass| L["Open-loop Poisson load generator<br/>or 24 h diurnal replay"]
    L -->|HTTP| S["Dynamic-batching server<br/>max_batch · 2 ms window"]
    S --> B["Backend: eager FP16 · torch.compile ·<br/>ORT CUDA · TensorRT FP16"]
    B --> G["GPU or MIG slice<br/>NVML power every 100 ms"]
    S --> R[("results/ JSONL<br/>every request")]
    R --> F["harness/report.py → REPORT.md,<br/>notebook, figures"]

Workload. A request is the 3-question bundle an agent fires per event (route: choice, 3 options; severity: score, 3 levels; needs_human: yes/no), over one of 1,000 public U.S. federal contract-opportunity notices sampled from SAM.gov's data.gov bulk extract (fixtures/workload/, provenance in source.json). Length buckets by each checkpoint's own tokenizer: short ≤ 80, medium 160–360, long 400–472, xlong 700–984 tokens.

Timing boundary. Shootout and Offline time prompt construction → tokenization → host-to-device copy → forward → device-to-host copy (which synchronizes) → temperature calibration → answer formatting; model load is excluded. This is the boundary laya-mlx's BENCHMARKS.md uses, so those rows are comparable. Server numbers are client-observed request latency over loopback HTTP, including queueing and batching. Power is board power sampled in-process with NVML every 100 ms (on a MIG slice this is the whole card).

Batching. Upstream Agent.predict batches the questions of one state. Here a batch is any set of (state, question) rows built with the upstream build_sequence, so tokens are identical, and the model is called directly (harness/sequences.py), the same way the MLX port produced its 64-batch throughput number.

Pinned. Upstream sources, weights (by SHA256) and library versions are in manifest.json and REPORT.md §2; each box's driver, CUDA, clocks and MIG state are in results/<box>/env.json.

./setup_env.sh .venv python3.12          # torch 2.14 cu130, laya @ pinned commit, ORT 1.30 (CUDA 13 feed), TensorRT 10.16
hf download convaiinnovations/laya --revision c5d78730f3493e4fe16d61507ef4b78eef7318cf --local-dir models/laya
hf download convaiinnovations/laya-multilingual --local-dir models/laya-multilingual
hf download convaiinnovations/laya-typed-decisions --local-dir models/laya-typed-decisions
python scripts/make_manifest.py          # compare the SHA256s with the committed manifest.json
python -m harness.envinfo --box mybox
python -m harness.export_onnx
python -m harness.parity --box mybox --backends eager-fp32 eager-fp16 ort-cuda-fp16 ort-trt-fp16 compile-fp16
python -m harness.shootout --box mybox
python -m harness.sweep --box mybox --model laya --backend eager-fp16
python -m harness.offline --box mybox --model laya --backend eager-fp16
python scripts/make_figures.py           # redraw docs/figures/ from results/

fixtures/workload/make_workload.py rebuilds the workload from the current SAM.gov extract; the committed states.jsonl is the frozen version used for every number here. notebooks/report.ipynb recomputes every table and chart from results/ (cost assumptions in its first cell).

harness/     parity, shootout, server + load generator + sweep, offline, baselines, report/cost model
fixtures/    parity cases with upstream expected outputs; the frozen SAM.gov workload and diurnal curve
results/     raw per-request / per-iteration JSONL and per-run JSON for every box (the record)
docs/figures/  README and report figures (regenerate with scripts/make_figures.py)
notebooks/   report.ipynb, all tables and charts from results/
k8s/, scripts/  the per-box run chains and the H100 MIG window checklist
REPORT.md    the full study: method, results, synthesis, dead ends, limitations, follow-ups

• Parity on 63 fixtures is fidelity to upstream, not task accuracy. No accuracy claim is made for any model.
• The Jev row is measured over the public internet from Arizona; its latency includes the network.
• MIG power is per physical card; per-slice energy is not measurable with NVML.
• The load generator runs on the same host over loopback; the serving stack is a ~120-line asyncio dynamic batcher, not Triton (same algorithm; Triton is a follow-up row).
• LLM baselines report latency/throughput/tokens only; one run per sweep configuration. The full list is in REPORT.md §10.

If you use the numbers, harness or workload, please cite (GitHub's "Cite this repository" button reads CITATION.cff):

@misc{kinge2026layacudabench,
  author       = {Kinge, Bhushan},
  title        = {laya-cuda-bench: Capacity, Tail Latency and Cost of the Laya Decision Models on NVIDIA GPUs},
  year         = {2026},
  howpublished = {\url{https://github.com/bhushankinge/laya-cuda-bench}}
}

Apache-2.0. Built on NandhaKishorM/laya and the convaiinnovations/laya weights, mizorewww/laya-mlx (parity fixtures and timing boundary) and receptron/laya (ONNX export reference). Workload text: public SAM.gov notices. Full attributions in NOTICE.