feat(evals): judge open-ended answers and record judge identity - #8574

Open

sudoKrishna wants to merge 23 commits into

simstudioai:mainfrom

sudoKrishna:feat/evals-judged-scenarios

Commits on Sep 29, 2026

feat(evals): add agent tool-use evaluation harness
sudoKrishna
committed
86c79d7
feat(evals): add live DeepSeek model runs to the agent eval suite
sudoKrishna
committed
af2e6ca
test(evals): assert grounded behavior instead of exact phrasing in live mode
sudoKrishna
committed
433fd0f
feat(evals): run agent scenarios through the DAGExecutor
sudoKrishna
committed
85525a4
test(evals): assert executor block retry on agent failure
sudoKrishna
committed
fb7f54c
test(evals): assert executor model fallback on primary failure
sudoKrishna
committed
c93426b
feat(evals): record and replay live model transcripts
sudoKrishna
committed
2032c4d
feat(evals): add agent context eval suite
sudoKrishna
committed
c12342a

Commits on Sep 30, 2026

feat(evals): compare agent tool-use across models
sudoKrishna
committed
26937ff

Commits on Oct 1, 2026

Commits on Oct 2, 2026

test(evals): address review findings on the eval harness
sudoKrishna
committed
f619479
feat(evals): judge open-ended agent answers
sudoKrishna
committed
3d8de59
feat(evals): record judge identity and refuse cross-identity comparisons
sudoKrishna
committed
74faa92
feat(evals): judge the remaining open-ended scenarios
sudoKrishna
committed
2023d07
fix(evals): address the judge review findings
sudoKrishna
committed
8bf9c01