Repository navigation
(bedrock-agentcore-alpha): add OnlineEvaluationConfig and Evaluator L2 constructs for online evaluation #37614
Description
Describe the feature
Add L2 constructs to the @aws-cdk/aws-bedrock-agentcore-alpha module for Amazon Bedrock AgentCore Online Evaluation. This includes:
OnlineEvaluationConfig— L2 construct backed byCfnOnlineEvaluationConfigthat configures continuous evaluation of agent traces. Auto-creates an IAM execution role with
least-privilege permissions, supports enabling/disabling evaluation, and accepts both built-in and custom evaluators.Evaluator— L2 construct backed byCfnEvaluatorfor defining custom evaluation logic using either LLM-as-a-Judge (model-based) or code-based (Lambda) evaluator configurations.EvaluatorReference— Unified entry point for referencing both built-in evaluators (e.g., HELPFULNESS, CORRECTNESS) and customEvaluatorconstructs.EvaluatorRatingScale— Factory class for categorical and numerical rating scales.DataSourceConfig— Configuration for evaluation data sources (CloudWatch Logs or AgentCore Runtime endpoints).
The constructs implement IGrantable and ITaggableV2, auto-wire IAM permissions (CloudWatch Logs, Bedrock model invocation, Lambda invoke with confused deputy prevention), and follow
CDK L2 design patterns including import/export, validation, and token safety.
Use Case
CDK users building agentic applications with Amazon Bedrock AgentCore need a way to configure online evaluation of their agent traces directly in their CDK applications. Currently there
is no L2 support for this — users would need to use L1 (CfnOnlineEvaluationConfig, CfnEvaluator) constructs and manually wire up IAM roles, permissions, and resource references.
With these L2 constructs, users can:
- Set up continuous evaluation of agent traces with sensible defaults and minimal boilerplate
- Use built-in evaluators (HELPFULNESS, CORRECTNESS, FAITHFULNESS, etc.) without understanding the underlying CloudFormation details
- Define custom evaluators using LLM-as-a-Judge or Lambda-based evaluation logic
- Have IAM permissions automatically scoped with least-privilege and confused deputy prevention
Proposed Solution
Implementation is ready in PR form. The design follows CDK L2 conventions:
- Separate files for each concern (construct, types, validation, permissions)
- Props-based API with strong CDK types and
@defaultdocumentation fromOnlineEvaluationConfigArn()andfromEvaluatorArn()static import methods- Auto-generated execution role with scoped IAM permissions
- 69 unit tests and an integration test with snapshot
Other Information
No response
Acknowledgements
- I may be able to implement this feature request
- This feature might incur a breaking change
AWS CDK Library version (aws-cdk-lib)
latest
AWS CDK CLI version
2.1108.0 (build eace286)
Environment details (OS name and version, etc.)
macOS 15.4.1
Activity
pahud commentedon Apr 17, 2026
Thanks for this feature request and the accompanying PR #37615. The implementation looks comprehensive — it follows the existing bedrock-agentcore-alpha patterns (interface + base class + concrete class) and includes thorough test coverage.
feat(bedrock-agentcore-alpha): add OnlineEvaluationConfig and Evaluat…
github-actions commentedon Apr 30, 2026
Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.
feat(bedrock-agentcore-alpha): add OnlineEvaluationConfig and Evaluat…