Skip to content

(bedrock-agentcore-alpha): add OnlineEvaluationConfig and Evaluator L2 constructs for online evaluation #37614

Closed

Description

@rezabekf

Describe the feature

Add L2 constructs to the @aws-cdk/aws-bedrock-agentcore-alpha module for Amazon Bedrock AgentCore Online Evaluation. This includes:

  • OnlineEvaluationConfig — L2 construct backed by CfnOnlineEvaluationConfig that configures continuous evaluation of agent traces. Auto-creates an IAM execution role with
    least-privilege permissions, supports enabling/disabling evaluation, and accepts both built-in and custom evaluators.
  • Evaluator — L2 construct backed by CfnEvaluator for defining custom evaluation logic using either LLM-as-a-Judge (model-based) or code-based (Lambda) evaluator configurations.
  • EvaluatorReference — Unified entry point for referencing both built-in evaluators (e.g., HELPFULNESS, CORRECTNESS) and custom Evaluator constructs.
  • EvaluatorRatingScale — Factory class for categorical and numerical rating scales.
  • DataSourceConfig — Configuration for evaluation data sources (CloudWatch Logs or AgentCore Runtime endpoints).

The constructs implement IGrantable and ITaggableV2, auto-wire IAM permissions (CloudWatch Logs, Bedrock model invocation, Lambda invoke with confused deputy prevention), and follow
CDK L2 design patterns including import/export, validation, and token safety.

Use Case

CDK users building agentic applications with Amazon Bedrock AgentCore need a way to configure online evaluation of their agent traces directly in their CDK applications. Currently there
is no L2 support for this — users would need to use L1 (CfnOnlineEvaluationConfig, CfnEvaluator) constructs and manually wire up IAM roles, permissions, and resource references.

With these L2 constructs, users can:

  • Set up continuous evaluation of agent traces with sensible defaults and minimal boilerplate
  • Use built-in evaluators (HELPFULNESS, CORRECTNESS, FAITHFULNESS, etc.) without understanding the underlying CloudFormation details
  • Define custom evaluators using LLM-as-a-Judge or Lambda-based evaluation logic
  • Have IAM permissions automatically scoped with least-privilege and confused deputy prevention

Proposed Solution

Implementation is ready in PR form. The design follows CDK L2 conventions:

  • Separate files for each concern (construct, types, validation, permissions)
  • Props-based API with strong CDK types and @default documentation
  • fromOnlineEvaluationConfigArn() and fromEvaluatorArn() static import methods
  • Auto-generated execution role with scoped IAM permissions
  • 69 unit tests and an integration test with snapshot

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

AWS CDK Library version (aws-cdk-lib)

latest

AWS CDK CLI version

2.1108.0 (build eace286)

Environment details (OS name and version, etc.)

macOS 15.4.1

Activity

  1. added
    feature-requestA feature should be added or improved.
    needs-triageThis issue or PR still needs to be triaged.
    on Apr 16, 2026
  2. pahud commented on Apr 17, 2026

    @pahud
    Contributor

    Thanks for this feature request and the accompanying PR #37615. The implementation looks comprehensive — it follows the existing bedrock-agentcore-alpha patterns (interface + base class + concrete class) and includes thorough test coverage.

  3. added
    effort/mediumMedium work item – several days of effort
    and removed
    needs-triageThis issue or PR still needs to be triaged.
    on Apr 17, 2026
  4. added a commit that references this issue on Apr 30, 2026
    c13de04
  5. github-actions commented on Apr 30, 2026

    @github-actions
    Contributor

    Comments on closed issues and PRs are hard for our team to see.
    If you need help, please open a new issue that references this one.

  6. locked as resolved and limited conversation to collaborators on Apr 30, 2026
  7. added a commit that references this issue on Jul 29, 2026
    128c4b3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    @aws-cdk/aws-iamRelated to AWS Identity and Access Managementeffort/mediumMedium work item – several days of effortfeature-requestA feature should be added or improved.p2

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions