Handle models missing from the pricing table by Copilot · Pull Request #66552 · github/gh-aw

Copilot · 2026-10-07T12:12:17Z

Models without AI-credit pricing can be rejected by the AWF API proxy at runtime. This change supplies known catalog prices during compilation and warns when a configured model has no price or fallback.

Price propagation: Inject exact local catalog prices into the compiled AWF pricing overlay, including gpt-6.1-sol.
Compile-time warning: For unpriced models, recommend adding model-specific pricing, configuring a fallback, mapping to a priced model, or choosing another model.

models:
  default-ai-credits-pricing:
    input: 0.000002
    output: 0.00001

Fixes [AW Top 10] 08 Handle models missing from the pricing table #66492

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot

Copilot review overview

🟡 Changes recommended

Parameterized models, glob-based aliases, and empty pricing entries can bypass propagation or produce incorrect diagnostics.

Review effort: Balanced
Findings: 3

Open (3)

Resolve alias glob targets against the pricing catalog · New
Require non-empty pricing data for model entries · New
Strip model query parameters before catalog lookup · New

What changed in this PR

Adds compile-time model pricing propagation and diagnostics to prevent AWF proxy failures for unpriced models.

Changes:

Registers exact catalog pricing resolution during compilation.
Warns when configured models lack pricing or fallback configuration.
Adds pricing propagation and warning tests.

File	Description
`pkg/workflow/compiler_types.go`	Clarifies pricing resolver behavior.
`pkg/workflow/compiler_mutators.go`	Exposes resolver configuration.
`pkg/workflow/compiler_model_pricing.go`	Updates pricing resolution documentation.
`pkg/cli/model_costs.go`	Adds exact catalog lookup.
`pkg/cli/compile_model_validation.go`	Adds missing-pricing diagnostics.
`pkg/cli/compile_development_test.go`	Tests pricing warnings.
`pkg/cli/compile_compiler_setup.go`	Registers pricing resolver and validator.
`pkg/cli/compile_compiler_setup_test.go`	Tests catalog-price injection.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Copilot · 2026-10-07T14:22:42Z

+		for _, target := range targets {
+			targetProvider, targetModel, ok := configuredModelPricingID(&workflow.WorkflowData{
+				EngineConfig: &workflow.EngineConfig{LLMProvider: workflow.LLMProvider(provider)},
+			}, target)
+			if !ok || !configuredModelHasPricing(targetProvider, targetModel, aliases, visited) {


Handled in b2742c01: pricing checks now match alias glob targets against priced catalog models, with built-in sonnet and haiku regression coverage.

Copilot · 2026-10-07T14:22:43Z

+		for name := range models {
+			if strings.EqualFold(name, model) {
+				return true


Handled in b2742c01: both compiler and workflow pricing predicates now require a non-empty cost map; regression tests cover absent and empty pricing entries.

Copilot · 2026-10-07T14:22:43Z

+	initModelPrices()
+
+	normalizedProvider := modelsdev.NormalizeProvider(provider)
+	comparableModel := modelsdev.NormalizeComparableModelID(model)


Handled in b2742c01: query parameters are removed before exact catalog lookup and before workflow overlay injection, with a ?effort=high regression test.

gh-aw-bot · 2026-10-07T13:53:57Z

@copilot address the following outstanding work in one pass:

Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
Review (pkg/cli/compile_model_validation.go:228): Alias targets can be glob patterns, but this loop treats each target as an exact model ID. Built-ins such as sonnet and haiku immediately fail on targets like copilot/*sonnet*, so ordinary valid aliases emit the new "no pricing" warning even though they resolve at runtime to catalog-priced models. Resolve patterns against the priced catalog using the same alias/glob semantics as the runtime, and add a built-in alias regression case. - Handle models missing from the pricing table #66552 (comment)
Review (pkg/cli/compile_model_validation.go:259): A model entry is not necessarily a pricing entry: the schema permits models.providers.openai.models.custom: {} (and an empty cost object), but this returns true solely from the key name and suppresses the required warning. Check for an actual non-empty cost map instead. The workflow-side modelCostsHasPricingFor predicate must be updated consistently as well, otherwise the catalog resolver will still skip injection for the same empty entry. - Handle models missing from the pricing table #66552 (comment)
Review (pkg/cli/model_costs.go:148): Strip model-identifier query parameters before the exact catalog lookup. The compiler-side resolver forwards WorkflowData.Model unchanged, so a valid value such as gpt-6.1-sol?effort=high misses this lookup. The warning path does strip the query and therefore considers the model priced, leaving the compiled overlay empty with no diagnostic; this can still produce the runtime HTTP 400 the PR is intended to prevent. - Handle models missing from the pricing table #66552 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 2732e85
Sous-chef work: 33407c824794f0c6847658dd00beb5816ef104553e0ef5b40986f6bf196e496d 87840d7cc6cd10302f381a39f07f9fd091729eb5d1a75e23eef8b6b0bbc34bf9 dc4c79fb260938d33fdb4b047fbbb3ec45d0dea344c826813f4e279b9ae7ee43
Sous-chef state: bb268c8ac2d9c1f7677871c1c5bcf18bf2fa4b9700619465ab5f5483035178c8

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 7.92 AIC · ⌖ 6.03 AIC · ⊞ 1K · ◷
_{Comment /souschef to run again}

github-actions · 2026-10-07T13:56:03Z

🔬 Test Quality Sentinel is analyzing test quality on this pull request...

github-actions · 2026-10-07T13:56:03Z

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

🏗️ ADR gate enforced by Design Decision Gate 🏗️

github-actions · 2026-10-07T13:57:13Z

🧠 Matt Pocock Skills Reviewer is reviewing this pull request using Matt Pocock's engineering skills...

github-actions · 2026-10-07T13:59:04Z

✅ PR Code Quality Reviewer completed the code quality review.

review submitted via overall review; existing inline comments already covered the other blockers

🔎 Code quality review by PR Code Quality Reviewer

github-actions · 2026-10-07T14:01:49Z

🏗️ Design Decision Gate — ADR Required

Result: ❌ No Architecture Decision Record found for this PR. A draft ADR has been generated and committed to this branch for your review.

Why this PR is gated

Check	Value
`implementation` label	not present
New lines in business-logic dirs (`pkg/`, ...)	270 (threshold: 100)
Existing ADR in PR body / branch / linked issue (#66492)	none found

Draft ADR

📄 docs/adr/66552-compile-time-model-pricing-injection-and-warning.md — Status: Draft

Decision captured from the diff:

Decision: always resolve model pricing from the local catalog during compilation (SetModelPricingResolver + the configured-model validator are now registered unconditionally in createAndConfigureCompiler), inject exact catalog prices into the AWF pricing overlay, and warn at compile time when a configured model has no price and no fallback.
Driver: the AWF API proxy rejects inference for unpriced models with HTTP 400 at run time (issue [AW Top 10] 08 Handle models missing from the pricing table #66492, sources [aw] GPL Dependency Cleaner (gpclean) has no AI credits pricing for model (gpt-6.1-sol) #66425 / [aw] Schema Consistency Checker has no AI credits pricing for model (gpt-6.1-sol) #66459); retries cannot help, so the check must move to compile time.
Alternatives recorded: (1) hard compilation error on unpriced models, (2) proxy-side default pricing / keep failing at run time, (3) reuse the existing prefix-based findModelPricing instead of the new exact-match findExactModelPricing.
Key trade-off: exact matching avoids silently mispricing gpt-6.1-sol-preview as gpt-6.1-sol, at the cost of false-positive warnings for model names the proxy knows but the local catalog does not.

Next action

Review the draft, correct anything the gate inferred incorrectly (especially the alternatives and the negative consequences), and change Status from Draft to Proposed/Accepted before merging.

🏗️ ADR gate enforced by Design Decision Gate 🏗️ · pi · opus50 · 42.1 AIC · ⌖ 50.4 AIC · ⊞ 1.7K · ◷
_{Comment /review to run again}

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

…dle-models-missing-pricing-table Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

github-actions · 2026-10-07T14:05:19Z

@

🧪 Test quality analysis by Test Quality Sentinel · copilot · haiku45 · 55.4 AIC · ⌖ 7.4 AIC · ⊞ 8.2K · ◷
_{Comment /review to run again}

github-actions

✅ Test Quality Sentinel: 84/100 (Excellent). 12 design tests (86%) vs 2 implementation tests (14%). No violations. Strong edge-case coverage with 28+ scenarios across table-driven tests. Comprehensive model pricing feature testing with 7-scenario pricing warning test, all error paths covered.

🧪 Test quality analysis by Test Quality Sentinel · copilot · haiku45 · 55.4 AIC · ⌖ 7.4 AIC · ⊞ 8.2K
_{Comment /review to run again}

github-actions

Impeccable review — `harden` + `audit` (bug-fix change)

This PR adds compile-time warnings for models missing AI-credits pricing. I verified the three edge-case correctness issues already flagged by the automated Copilot review are still present and reproducible against the current diff:

Built-in alias glob targets (sonnet, haiku, etc.) trigger false-positive warnings — configuredModelHasPricing treats alias targets like copilot/*sonnet* as literal model IDs, so ordinary workflows using built-in aliases will see a spurious "no AI credits pricing" warning.
Query-parameter model identifiers (e.g. gpt-6.1-sol?effort=high) are not stripped in findExactModelPricing, so valid Model-Alias-Format identifiers silently miss catalog pricing.
hasModelCostOverlay returns true for a frontmatter model entry with no actual cost, suppressing the new warning exactly in the case it's meant to catch.

All three were reproduced locally with small standalone test snippets (see inline comments for exact repro code). These are correctness bugs in the new warning logic itself — the warning will misfire (false positives for common built-in aliases) and under-fire (false negatives for stub entries), undermining the PR's stated goal of catching unpriced models before the AWF API proxy rejects them at runtime.

Requesting changes to fix these before merge.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 69.3 AIC · ⌖ 15.8 AIC · ⊞ 8.1K

github-actions · 2026-10-07T14:07:15Z

+		for _, target := range targets {
+			targetProvider, targetModel, ok := configuredModelPricingID(&workflow.WorkflowData{
+				EngineConfig: &workflow.EngineConfig{LLMProvider: workflow.LLMProvider(provider)},
+			}, target)


Alias glob patterns break configuredModelHasPricing and emit false-positive warnings for every built-in model alias.

Built-in aliases such as sonnet and haiku resolve to glob targets like copilot/*sonnet* (see pkg/workflow/model_aliases.go), but configuredModelPricingID (and the exact-match lookup it feeds into) treats each target as a literal model ID. A glob pattern never matches any catalog record, so configuredModelHasPricing returns false and the new warning fires for every ordinary workflow using a built-in alias, even though AWF resolves these at runtime.

Reproduced locally:

aliases := workflow.MergeImportedModelAliases(nil, nil) data := &workflow.WorkflowData{Model: "sonnet", EngineConfig: &workflow.EngineConfig{ID: "copilot"}, ModelMappings: aliases} configuredModelPricingWarning(data) // => "Model sonnet has no AI credits pricing..."

This fires spuriously for any workflow with model: sonnet or model: haiku, which is likely very common.

Please either skip/short-circuit glob targets (treat them as "has pricing" since AWF resolves them dynamically) or expand the glob against the catalog before declaring no pricing. @copilot please address this.

Handled in b2742c01: alias glob targets are matched against priced catalog models, and built-in sonnet/haiku cases are covered by tests.

github-actions · 2026-10-07T14:07:15Z

+	initModelPrices()
+
+	normalizedProvider := modelsdev.NormalizeProvider(provider)
+	comparableModel := modelsdev.NormalizeComparableModelID(model)


Query-parameter suffixes are not stripped before the exact-pricing lookup.

configuredModelPricingID strips a ?effort=... suffix from the configured model before calling findExactModelPricing, but the pricing resolver wired up in compile_compiler_setup.go (findExactModelPricing) is also called directly by resolveModelPricingIfMissing via the injected modelPricingResolver, and findExactModelPricing itself does not strip query parameters from model.

Reproduced locally:

findExactModelPricing("openai", "gpt-6.1-sol?effort=high") // => (nil, false) findExactModelPricing("openai", "gpt-6.1-sol") // => (pricing, true)

Any workflow configuring model: openai/gpt-6.1-sol?effort=high (a valid Model Alias Format identifier per model_alias_validation.go) will silently lose the catalog pricing match. Strip the query suffix (e.g. strings.Cut(model, "?")) at the top of findExactModelPricing, mirroring what configuredModelPricingID already does. @copilot please address this.

Handled in b2742c01: the workflow resolver and exact catalog lookup strip query parameters; regression coverage verifies the base model is injected.

github-actions · 2026-10-07T14:07:15Z

+		}
+		for name := range models {
+			if strings.EqualFold(name, model) {
+				return true


hasModelCostOverlay returns true for an entry with no cost field, suppressing the new pricing warning incorrectly.

The schema permits a frontmatter model entry such as models.providers.openai.models.custom: {} (no cost key, or an empty cost map). hasModelCostOverlay only checks that the model key exists under providerMap["models"], not that it actually carries pricing data:

costs := map[string]any{"providers": map[string]any{"openai": map[string]any{"models": map[string]any{"custom-model": map[string]any{}}}}} hasModelCostOverlay(costs, "openai", "custom-model") // => true (should be false — no cost present)

This means a frontmatter stub entry with no actual price silently suppresses configuredModelPricingWarning, defeating the purpose of this PR (catching models the AWF proxy will reject with HTTP 400). Check that cost is present and non-empty before returning true. @copilot please address this.

Handled in b2742c01: an overlay only counts as pricing when the model has a non-empty cost map, covered for missing and empty maps.

github-actions

Request changes

The new pricing-warning path still misclassifies several valid model configurations and misses some genuinely unpriced ones, so it is not safe to ship yet.

Blocking themes

Parameterized model IDs such as gpt-6.1-sol?effort=high miss the new exact catalog lookup, so compilation still omits pricing for a model the local catalog already knows.
Alias handling is too strict for documented model maps: glob-backed builtins like sonnet/haiku are treated as unpriced, and same-workflow aliases are not even present in WorkflowData.ModelMappings during this check.
The overlay check only looks for a matching model key, so placeholder entries without usable cost data suppress the warning even though the AWF proxy can still reject the model at runtime.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 91.9 AIC · ⌖ 6.68 AIC · ⊞ 19.6K
_{Comment /review to run again}

github-actions · 2026-10-07T14:07:30Z

+	if !ok || hasModelCostOverlay(data.ModelCosts, provider, model) {
+		return ""
+	}
+	if configuredModelHasPricing(provider, model, data.ModelMappings, make(map[string]struct{})) {


This alias pricing check depends on WorkflowData.ModelMappings, but that field is populated from imports only, so a workflow that defines models.preferred-model: [gpt-6.1-sol] in its own frontmatter still trips the new has no AI credits pricing warning.
💡 Why this blocks
workflow_builder.go currently calls MergeImportedModelAliases(importsResult.MergedModels, nil), so same-file aliases never reach this helper. Merge the main frontmatter aliases into WorkflowData.ModelMappings, or pass the parsed frontmatter alias map into this check instead of relying on the imported-only field.

I did not change this: workflow aliases under models are not supported by the current schema (main_workflow_schema.json:3055-3104 sets additionalProperties: false), and the existing import test documents that aliases now come only from the built-in map (model_aliases_import_test.go:18-20). Such frontmatter is rejected before the pricing warning runs.

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

pelikhan · 2026-10-07T15:49:34Z

@copilot run pr-finisher skill

…dle-models-missing-pricing-table # Conflicts: # .github/workflows/agentic-token-trend-audit.lock.yml # .github/workflows/audit-workflows.lock.yml # .github/workflows/avenger.lock.yml # .github/workflows/changeset.lock.yml # .github/workflows/cloclo.lock.yml # .github/workflows/commit-changes-analyzer.lock.yml # .github/workflows/copilot-centralization-optimizer.lock.yml # .github/workflows/daily-awf-spec-compiler-surfacing.lock.yml # .github/workflows/daily-cli-performance.lock.yml # .github/workflows/daily-doc-updater.lock.yml # .github/workflows/daily-evals-report.lock.yml # .github/workflows/daily-go-test-parallelizer.lock.yml # .github/workflows/daily-regulatory.lock.yml # .github/workflows/daily-spending-forecast.lock.yml # .github/workflows/daily-windows-defender-scan.lock.yml # .github/workflows/dependabot-go-checker.lock.yml # .github/workflows/eslint-monster.lock.yml # .github/workflows/front-page-copy-guard.lock.yml # .github/workflows/github-remote-mcp-auth-test.lock.yml # .github/workflows/grumpy-reviewer.lock.yml # .github/workflows/issue-triage-agent.lock.yml # .github/workflows/metrics-collector.lock.yml # .github/workflows/outcome-collector.lock.yml # .github/workflows/poem-bot.lock.yml # .github/workflows/purelock.lock.yml # .github/workflows/repo-audit-analyzer.lock.yml # .github/workflows/schema-feature-coverage.lock.yml # .github/workflows/sighthound-security-scan.lock.yml # .github/workflows/smoke-otel-backends.lock.yml # .github/workflows/update-astro.lock.yml # .github/workflows/windows.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-10-07T16:13:51Z

@copilot address the following outstanding work in one pass:

Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

The three requested pricing fixes and regression cases were already present in b2742c0; I verified them, merged the latest main, and regenerated the workflow locks in merge commit 2a75a62e. The final local gate passed. The existing replies on the three review comments address their findings; this environment does not expose a tool to resolve review threads.

done

Initial plan

e5fbb5a

Copilot AI assigned Copilot and pelikhan Oct 7, 2026

Copilot started work on behalf of pelikhan October 7, 2026 12:12 View session

Copilot AI linked an issue Oct 7, 2026 that may be closed by this pull request

[AW Top 10] 08 Handle models missing from the pricing table #66492

Closed

2 tasks

Inject model pricing and warn on misses

2732e85

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix models missing pricing in the pricing table~~ Handle models missing from the pricing table Oct 7, 2026

Copilot finished work on behalf of pelikhan October 7, 2026 12:37

Copilot AI requested a review from pelikhan October 7, 2026 12:37

pelikhan marked this pull request as ready for review October 7, 2026 13:28

Copilot AI balanced review requested due to automatic review settings October 7, 2026 13:28

Copilot started reviewing on behalf of pelikhan October 7, 2026 13:28 View session

Copilot AI reviewed Oct 7, 2026

View reviewed changes

github-actions Bot mentioned this pull request Oct 7, 2026

[pr-sous-chef] 🍳 PR Sous Chef Summary: 4 PRs nudged toward completion #66387

Closed

Copilot started work on behalf of gh-aw-bot October 7, 2026 13:54 View session

docs: add draft ADR for compile-time model pricing injection

c4751e5

Copilot AI and others added 2 commits October 7, 2026 14:03

Regenerate workflow pricing overlays

89f9659

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into copilot/aw-top-10-han…

46f8220

…dle-models-missing-pricing-table Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

github-actions Bot approved these changes Oct 7, 2026

View reviewed changes

github-actions Bot requested changes Oct 7, 2026

View reviewed changes

github-actions Bot previously requested changes Oct 7, 2026

View reviewed changes

This was referenced Oct 7, 2026

[aw] Test Quality Sentinel reported incomplete result #66592

Closed

[aw] Matt Pocock Skills Reviewer timed out #66595

Closed

Fix model pricing review findings

b2742c0

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot finished work on behalf of gh-aw-bot October 7, 2026 14:26

Copilot AI requested a review from gh-aw-bot October 7, 2026 14:26

Copilot started work on behalf of pelikhan October 7, 2026 15:50 View session

Copilot finished work on behalf of pelikhan October 7, 2026 16:14

pelikhan closed this Oct 7, 2026

Conversation

Copilot AI commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Copilot AI Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

gh-aw-bot commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 7, 2026

🏗️ Design Decision Gate — ADR Required

Why this PR is gated

Draft ADR

Next action

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Impeccable review — harden + audit (bug-fix change)

Uh oh!

github-actions Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Request changes

Uh oh!

github-actions Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

pelikhan commented Oct 7, 2026

Uh oh!

Copilot AI commented Oct 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Copilot AI commented Oct 7, 2026 •

edited

Loading

github-actions Bot commented Oct 7, 2026 •

edited

Loading

github-actions Bot commented Oct 7, 2026 •

edited

Loading

Impeccable review — `harden` + `audit` (bug-fix change)