Skip to content

feat(evals): add adversarial agent tool-use scenarios #8512

Description

@sudoKrishna

Problem

The tool-use suite passes 100% on six easy cases, which measures little. The
behaviors that actually break agents are disambiguation, refusal, empty results,
and long dependency chains. None are covered.

Proposal

Add five adversarial scenarios to agent-tool-use/scenarios.ts:

  • no-tool-needed — answer directly; call nothing even though tools exist
  • disambiguate-similar-tools — forecast tool vs current-weather tool
  • empty-result-no-hallucination — empty search; must not fabricate a policy
  • long-chain-dependency — four tools, each depending on the previous
  • near-duplicate-names — get_user vs get_user_settings

Scripted expectations keep them deterministic in CI; the same cases run in the
live and model-comparison suites.

Acceptance criteria

  • Five adversarial scenarios added and passing in the scripted suite
  • Each names the failure mode it targets
  • Live/model-comparison runs include them
  • Report shows the new cases

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions