Problem
The eval suite covers the tool loop, the executor's single-block paths, context
assembly, and model comparison. It does not cover orchestration: a parent
workflow invoking a child, parallel fan-out and merge, or a loop subagent. Those
paths are where result propagation, partial failure, and depth limits go wrong,
and nothing evaluates them end to end.
Proposal
Add an agent-orchestration suite that runs parent → child workflows through the
real DAGExecutor.
- Parent:
Start → Workflow block that invokes a named child.
- Child:
readWorkflowDefinitionAsExecutor is mocked to return a child
definition (e.g. Start → Agent, provider mocked); the child executor runs for
real, so the parent→child boundary is exercised, not simulated.
- Reuse the central
@sim/testing mocks already used by
executor/handlers/workflow/workflow-handler.test.ts (logging session,
billing attribution, permissions, auth-internal, snapshot, custom-block ops).
First scenarios
- Child output reaches the parent — the child returns a value; the parent's
Workflow block output maps it; the run succeeds.
- Child failure fails the parent — the child errors; the parent surfaces the
mapped error and does not report success.
- Call-chain depth limit — a recursive child chain stops at the limit
instead of looping forever.
Scope / difficulty
The Workflow block handler mixes persistence, billing, logging sessions,
delegation tokens, and custom-block policy; its unit test mocks ~15
collaborators. This eval should cover only the DAG-level integration the unit
test cannot, and reuse the central mocks rather than re-deriving them.
Acceptance criteria
Problem
The eval suite covers the tool loop, the executor's single-block paths, context
assembly, and model comparison. It does not cover orchestration: a parent
workflow invoking a child, parallel fan-out and merge, or a loop subagent. Those
paths are where result propagation, partial failure, and depth limits go wrong,
and nothing evaluates them end to end.
Proposal
Add an
agent-orchestrationsuite that runs parent → child workflows through thereal
DAGExecutor.Start→Workflowblock that invokes a named child.readWorkflowDefinitionAsExecutoris mocked to return a childdefinition (e.g.
Start → Agent, provider mocked); the child executor runs forreal, so the parent→child boundary is exercised, not simulated.
@sim/testingmocks already used byexecutor/handlers/workflow/workflow-handler.test.ts(logging session,billing attribution, permissions, auth-internal, snapshot, custom-block ops).
First scenarios
Workflow block output maps it; the run succeeds.
mapped error and does not report success.
instead of looping forever.
Scope / difficulty
The Workflow block handler mixes persistence, billing, logging sessions,
delegation tokens, and custom-block policy; its unit test mocks ~15
collaborators. This eval should cover only the DAG-level integration the unit
test cannot, and reuse the central mocks rather than re-deriving them.
Acceptance criteria
DAGExecutor, not the handler in isolation