Skip to content

feat(evals): evaluate subagent orchestration (parent → child workflow) #8570

Description

@sudoKrishna

Problem

The eval suite covers the tool loop, the executor's single-block paths, context
assembly, and model comparison. It does not cover orchestration: a parent
workflow invoking a child, parallel fan-out and merge, or a loop subagent. Those
paths are where result propagation, partial failure, and depth limits go wrong,
and nothing evaluates them end to end.

Proposal

Add an agent-orchestration suite that runs parent → child workflows through the
real DAGExecutor.

  • Parent: Start → Workflow block that invokes a named child.
  • Child: readWorkflowDefinitionAsExecutor is mocked to return a child
    definition (e.g. Start → Agent, provider mocked); the child executor runs for
    real, so the parent→child boundary is exercised, not simulated.
  • Reuse the central @sim/testing mocks already used by
    executor/handlers/workflow/workflow-handler.test.ts (logging session,
    billing attribution, permissions, auth-internal, snapshot, custom-block ops).

First scenarios

  1. Child output reaches the parent — the child returns a value; the parent's
    Workflow block output maps it; the run succeeds.
  2. Child failure fails the parent — the child errors; the parent surfaces the
    mapped error and does not report success.
  3. Call-chain depth limit — a recursive child chain stops at the limit
    instead of looping forever.

Scope / difficulty

The Workflow block handler mixes persistence, billing, logging sessions,
delegation tokens, and custom-block policy; its unit test mocks ~15
collaborators. This eval should cover only the DAG-level integration the unit
test cannot, and reuse the central mocks rather than re-deriving them.

Acceptance criteria

  • A parent invokes a child and the child's output reaches the parent result
  • A failed child fails the parent with the mapped error
  • Runs through the real DAGExecutor, not the handler in isolation
  • Results appear in a report and the README documents the suite
  • A negative check proves the assertions fail when child output is dropped

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions