Skip to content

feat(evals): judge open-ended answers and record judge identity #8574

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sudoKrishna wants to merge 24 commits into from
Closed

feat(evals): judge open-ended answers and record judge identity #8574

Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
86c79d7
feat(evals): add agent tool-use evaluation harness
sudoKrishna Sep 29, 2026
af2e6ca
feat(evals): add live DeepSeek model runs to the agent eval suite
sudoKrishna Sep 29, 2026
433fd0f
test(evals): assert grounded behavior instead of exact phrasing in li…
sudoKrishna Sep 29, 2026
85525a4
feat(evals): run agent scenarios through the DAGExecutor
sudoKrishna Sep 29, 2026
fb7f54c
test(evals): assert executor block retry on agent failure
sudoKrishna Sep 29, 2026
c93426b
test(evals): assert executor model fallback on primary failure
sudoKrishna Sep 29, 2026
2032c4d
feat(evals): record and replay live model transcripts
sudoKrishna Sep 29, 2026
c12342a
feat(evals): add agent context eval suite
sudoKrishna Sep 29, 2026
26937ff
feat(evals): compare agent tool-use across models
sudoKrishna Sep 30, 2026
a22387d
feat(evals): report failed checks in the model comparison
sudoKrishna Oct 1, 2026
9d2ec19
feat(evals): add adversarial agent tool-use scenarios
sudoKrishna Oct 1, 2026
cdebc4d
test(evals): fix adversarial scenario design after the live run
sudoKrishna Oct 1, 2026
fc549db
test(evals): fix chain and near-duplicate scenarios for live models
sudoKrishna Oct 1, 2026
4590af8
feat(evals): add LLM-as-judge scoring
sudoKrishna Oct 1, 2026
031f5ab
Merge branch 'feat/evals-agent-context' into feat/evals-integration
sudoKrishna Oct 1, 2026
ae57d49
Merge branch 'feat/evals-model-compare' into feat/evals-integration
sudoKrishna Oct 1, 2026
6dace97
Merge branch 'feat/evals-adversarial-scenarios' into feat/evals-integ…
sudoKrishna Oct 1, 2026
5235110
Merge branch 'feat/evals-llm-judge' into feat/evals-integration
sudoKrishna Oct 1, 2026
f619479
test(evals): address review findings on the eval harness
sudoKrishna Oct 2, 2026
3d8de59
feat(evals): judge open-ended agent answers
sudoKrishna Oct 2, 2026
74faa92
feat(evals): record judge identity and refuse cross-identity comparisons
sudoKrishna Oct 2, 2026
2023d07
feat(evals): judge the remaining open-ended scenarios
sudoKrishna Oct 2, 2026
8bf9c01
fix(evals): address the judge review findings
sudoKrishna Oct 2, 2026
69af5d7
feat(evals): record and replay judge transcripts
sudoKrishna Oct 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
test(evals): assert grounded behavior instead of exact phrasing in li… 
…ve mode

The first live DeepSeek run exposed brittle assertions, not harness bugs:
the model chained the tools correctly but the checks were case-sensitive and
required an internal order id. Match the retrieved value case-insensitively
and let live runs accept the grounded status rather than the internal id.
  • Loading branch information
@sudoKrishna
sudoKrishna committed Sep 29, 2026
commit 433fd0f81f7f19425ea838a914136489eda4ffe1
6 changes: 4 additions & 2 deletions apps/sim/evals/agent-tool-use/scenarios.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ export const AGENT_TOOL_USE_SCENARIOS: AgentToolUseScenario[] = [
],
expect: {
toolCallSequence: ['list_files', 'read_file'],
finalContent: 'ship the eval harness',
finalContent: /ship the eval harness/i,
maxIterations: 4,
successfulToolCalls: 2,
},
Expand Down Expand Up @@ -216,6 +216,8 @@ export const AGENT_TOOL_USE_SCENARIOS: AgentToolUseScenario[] = [
finalContent: /A-1937.*shipped/,
maxIterations: 3,
},
/** A live model may answer with the user-facing order number and the grounded status. */
liveExpect: { finalContent: /shipped/i },
Comment thread
sudoKrishna marked this conversation as resolved.
Outdated
},
{
id: 'parallel-independent-tools',
Expand Down Expand Up @@ -245,7 +247,7 @@ export const AGENT_TOOL_USE_SCENARIOS: AgentToolUseScenario[] = [
],
expect: {
toolCallSequence: ['get_weather', 'get_news'],
finalContent: /12°C.*transit strike ends/,
finalContent: /12°C[\s\S]*transit strike ends/i,
maxIterations: 3,
successfulToolCalls: 2,
},
Expand Down