Issues

• ContextRelevancy divides by context length with no zero guard: an empty context gives NaN (serialized as the "skipped" null) or a perfect 1 in JS, and ZeroDivisionError in Python

  braintrustdata/autoevals#223 · Filter by author shaurya416 opened on Sep 25, 2026

• Cross-trial consistency (kappa, self-agreement) falls outside the Scorer shape

  braintrustdata/autoevals#219 · Filter by author palamim opened on Sep 7, 2026

• OrcaRouter provider support for Autoevals

  braintrustdata/autoevals#218 · Filter by author lovejones2914-spec opened on Sep 4, 2026

• closed_q_a.yaml drops OpenAI’s scoping clause; factuality.yaml has a traceable typo

  braintrustdata/autoevals#217 · Filter by author idilgozel opened on Aug 31, 2026

• Proposal: EvalPort adapter for autoevals' Scorer/Score primitives

  braintrustdata/autoevals#215 · Filter by author adhabnr-ux opened on Aug 29, 2026

• EmbeddingSimilarity class-level embedding cache ignores model and prefix — cross-model cache poisoning

  braintrustdata/autoevals#211 · Filter by author spacesheepinternet opened on Aug 4, 2026

• Proposal: opt-in pre-registration pattern for autoevals runs (PRML)

  braintrustdata/autoevals#206 · Filter by author sk8ordie84 opened on Jul 22, 2026
  1

• isWrapped crashes on custom OpenAI-compatible clients when the braintrust SDK is loaded (TypeError: reading 'completions')

  braintrustdata/autoevals#205 · Filter by author Jamie-Ellis opened on Jul 16, 2026

• Is the returned ExactMatch Score object the right minimal external-consumer seam?

  braintrustdata/autoevals#187 · Filter by author Rul1an opened on Apr 24, 2026
  3

• ContextPrecision does not meet the specification

  braintrustdata/autoevals#176 · Filter by author fizcogar opened on Feb 25, 2026
  2

• AnswerRelevancy uses model parameter instead of embedding_model for Embedding call

  braintrustdata/autoevals#166 · Filter by author cgoer opened on Jan 15, 2026