Make smoke token telemetry checks engine-aware by Copilot · Pull Request #65403 · github/gh-aw

Copilot · 2026-10-03T19:08:40Z

Kiro, Goose, and Cursor smoke workflows can succeed without proxy token records, but the shared check_token_telemetry job treats missing records as a failure.

Engine-aware checks: Report proxy token assertions as not applicable for Kiro, Goose, and Cursor. Retain both token assertions for engines that provide proxy counts.
Artifact handling: Fail the job if the agent artifact cannot be downloaded, rather than silently skipping its assertions.
Compiled workflows: Update importing lock files and add regression coverage for the engine-specific conditions.

Fixes [AW Top 10] 04 Fix check_token_telemetry for Kiro, Goose, Cursor #65375

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-10-03T19:47:12Z

✅ PR Code Quality Reviewer completed the code quality review.

Unable to emit PR review actions from this environment after a safeoutputs write permission failure.

🔎 Code quality review by PR Code Quality Reviewer

github-actions · 2026-10-03T19:47:17Z

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

github-actions · 2026-10-03T19:47:21Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

github-actions · 2026-10-03T19:47:26Z

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

🏗️ ADR gate enforced by Design Decision Gate 🏗️

github-actions · 2026-10-03T19:47:37Z

✅ Ponytail Reviewer completed successfully!

Lean already. Ship.

Generated by Ponytail Reviewer for #65403

Copilot

Copilot review overview

🟡 Changes recommended

The unrelated skill edit removes explicit routing for durable work-queue requests.

Review effort: Balanced
Findings: 1

Open (1)

Restore routing for durable dispatcher/worker queue requests · New

What changed in this PR

Makes shared smoke-test telemetry checks engine-aware while ensuring missing agent artifacts fail explicitly.

Changes:

Skips proxy-token assertions for Kiro, Goose, and Cursor with an informational notice.
Regenerates all importing workflow locks.
Adds regression coverage for engine conditions and artifact handling.

File	Description
`.github/workflows/shared/token-telemetry-check.md`	Adds engine-aware assertions and strict artifact downloading.
`pkg/workflow/token_telemetry_check_test.go`	Tests generated telemetry checks across four engines.
`.github/workflows/smoke-test-tools.lock.yml`	Regenerates imported telemetry job.
`.github/workflows/smoke-temporary-id.lock.yml`	Regenerates imported telemetry job.
`.github/workflows/smoke-service-ports.lock.yml`	Regenerates imported telemetry job.
`.github/workflows/smoke-project.lock.yml`	Regenerates imported telemetry job.
`.github/workflows/smoke-opencode.lock.yml`	Regenerates imported telemetry job.
`.github/workflows/smoke-kiro.lock.yml`	Skips unsupported proxy assertions for Kiro.
`.github/workflows/smoke-goose.lock.yml`	Skips unsupported proxy assertions for Goose.
`.github/workflows/smoke-gemini.lock.yml`	Regenerates imported telemetry job.
`.github/workflows/smoke-deepseek-harness.lock.yml`	Regenerates imported telemetry job.
`.github/workflows/smoke-cursor.lock.yml`	Skips unsupported proxy assertions for Cursor.
`.github/workflows/smoke-crush.lock.yml`	Regenerates imported telemetry job.
`.github/workflows/smoke-copilot.lock.yml`	Regenerates imported telemetry job.
`.github/workflows/smoke-copilot-arm.lock.yml`	Regenerates imported telemetry job.
`.github/workflows/smoke-copilot-aoai-entra.lock.yml`	Regenerates imported telemetry job.
`.github/workflows/smoke-copilot-aoai-apikey.lock.yml`	Regenerates imported telemetry job.
`.github/workflows/smoke-codex.lock.yml`	Regenerates imported telemetry job.
`.github/workflows/smoke-claude.lock.yml`	Regenerates imported telemetry job.
`.github/skills/agentic-workflows/SKILL.md`	Reorders work-queue guidance but drops its routing entry.

 - `.github/aw/update-agentic-workflow.md`
 - `.github/aw/upgrade-agentic-workflows.md`
 - `.github/aw/visual-regression.md`
+- `.github/aw/work-queue.md`


github-actions · 2026-10-03T19:52:07Z

Comment Memory

reviewed_at: 2026-10-03T19:48:39.803+00:00
review_event: COMMENT
top_themes:
  - brittle lockfile-format regression test
files_reviewed:
  - .github/workflows/shared/token-telemetry-check.md
  - pkg/workflow/token_telemetry_check_test.go
  - .github/workflows/smoke-kiro.lock.yml
  - .github/workflows/smoke-copilot.lock.yml
comment_count: 1

Note

This comment is managed by comment memory.

It stores persistent context for this thread in the code block at the top of this comment.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 41 AIC · ⌖ 7.12 AIC · ⊞ 20.2K · ◷
_{Comment /review to run again}

github-actions

Verdict

COMMENT — the engine-aware gating looks reasonable, but the new regression test is brittle because it asserts exact generated YAML formatting and job adjacency instead of the check_token_telemetry semantics.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 41 AIC · ⌖ 7.12 AIC · ⊞ 20.2K
_{Comment /review to run again}

Comments that could not be inline-anchored

pkg/workflow/token_telemetry_check_test.go:109

This test is coupled to the current lockfile formatting, so a harmless job reorder or YAML emitter change will fail CI even when the telemetry behavior is still correct.

<details><summary>💡 Why this matters</summary>

strings.SplitN(..., " conclusion: ", 2) assumes conclusion is always the next job, and the later Contains/Count checks depend on exact whitespace and list ordering. That makes the regression coverage noisy because unrelated compile changes can break the test.

A more s…

github-actions

Skills-Based Review 🧠

Applied /codebase-design and /tdd — the core fix is sound (engine-aware conditionals, hard artifact-download failure, regression test), but flagged maintainability gaps plus one scope-creep item.

📋 Key Themes & Highlights

Key Themes

Duplicated magic list: the ["kiro","goose","cursor"] unsupported-engine list is a raw string literal repeated 3× in token-telemetry-check.md with no single source of truth, instead of leveraging the existing EngineCapabilities/EnginesWithCapability registry pattern used elsewhere in the compiler.
Brittle regression test: token_telemetry_check_test.go asserts on exact YAML string/indentation snippets from generated lock files rather than compiling and inspecting structured output, making it fragile to incidental formatting changes.
Unrelated change: the SKILL.md reordering of work-queue.md is out of scope for this PR's stated purpose.

Positive Highlights

✅ Removing continue-on-error: true on the artifact download so missing artifacts now fail loudly instead of silently skipping assertions — directly fixes the regression described in the PR body.
✅ Good root-cause framing: distinguishing "no proxy telemetry for this engine" (expected, now reported via ::notice::) from "telemetry emitter broken" (real failure).
✅ New regression test exercises all 4 engines (kiro, goose, cursor, copilot) and would have caught the original bug.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 113.2 AIC · ⌖ 13.6 AIC · ⊞ 10K
_{Comment /matt to run again}

github-actions · 2026-10-03T19:57:40Z


      - name: Assert token_usage.jsonl is non-empty
-        if: steps.download-agent.outcome == 'success'
+        if: steps.download-agent.outcome == 'success' && !contains(fromJSON('["kiro","goose","cursor"]'), needs.activation.outputs.engine_id)


[/codebase-design] The set of engines without proxy token telemetry (["kiro","goose","cursor"]) is duplicated as a raw string literal 3 times in this file, with no single source of truth. If a new engine lacking proxy telemetry is added, or kiro/goose/cursor later gain telemetry, every occurrence must be updated in lockstep and the compiled lock files regenerated — easy to miss.

💡 Suggested approach

The codebase already has an EngineCapabilities struct (pkg/workflow/agentic_engine.go) and EnginesWithCapability(predicate) registry helper used to build "registry-driven lists of supported engines" elsewhere. Consider adding a ProxyTokenTelemetry bool capability field, then have the compiler inject the resulting engine-ID list into this shared workflow template (e.g. via a compiler-rendered {{ ... }} placeholder) instead of hardcoding it in the .md source. This keeps the list correct automatically as engines evolve, matching the stated purpose of EnginesWithCapability.

@copilot please address this.

github-actions · 2026-10-03T19:57:40Z

+- `.github/aw/work-queue.md`
 - `.github/aw/workflow-constraints.md`
 - `.github/aw/workflow-editing.md`
 - `.github/aw/workflow-patterns.md`


[/grill-with-docs] This skills-ordering tweak (reordering work-queue.md and adding it to the "After loading" bullet list) is unrelated to the PR's stated purpose of making smoke token telemetry checks engine-aware.

💡 Suggested approach

Unrelated doc/skill changes bundled into a focused bug-fix PR make the diff harder to review and the git history harder to bisect later. If this was an unintentional rebase artifact, drop it from this PR; if intentional, split it into its own PR so the token-telemetry fix stays a clean, reviewable unit.

@copilot please address this.

github-actions · 2026-10-03T19:57:40Z

+				"Report engines without proxy token telemetry",
+			} {
+				if !strings.Contains(check, expected) {
+					t.Errorf("token telemetry check missing %q", expected)


[/tdd] This test asserts on exact literal YAML substrings (indentation, step names, strings.Count on a hardcoded expression string) extracted from generated .lock.yml files rather than compiling the shared .md source directly and inspecting structured output. Any cosmetic change to the compiler's YAML formatting (e.g. indentation width, step ordering) will break this test even though behavior is unchanged.

💡 Suggested approach

Prefer compiling token-telemetry-check.md in-memory (as other shared-component tests do, e.g. TestCheckoutImportFromSharedWorkflow in checkout_import_test.go) and asserting on the compiler's own job/step model, or at minimum parse the YAML into a struct and assert on needs, if, and step names/conditions as data rather than raw string contains/counts. This decouples the regression test from incidental formatting of 15+ generated lock files.

@copilot please address this.

github-actions

Reviewed with Impeccable audit + harden modes (infra/CI change, no UI).

Summary: The engine-aware gating logic is correct and consistent across all 19 regenerated lock files. Verified:

All 17 workflows importing token-telemetry-check.md already define an activation job, so the new needs: [agent, activation] dependency is safe everywhere.
continue-on-error: true on the artifact download is intentionally removed per the PR description (fail fast instead of silently skipping assertions) — correct behavior change.
contains(fromJSON('["kiro","goose","cursor"]'), needs.activation.outputs.engine_id) correctly gates both proxy assertions (negated) and the new notice step (non-negated), and engine_id is a real activation job output (steps.generate_aw_info.outputs.engine_id).
Recompiling smoke-kiro.md reproduces the committed lock file byte-for-byte, confirming no stale/drifted lock files.
New Go test TestSmokeTokenTelemetryCheckIsEngineAware passes and correctly asserts the gating/notice/no-continue-on-error invariants across kiro, goose, cursor, and copilot.

No blocking issues found. Minor non-blocking note: the ["kiro","goose","cursor"] engine list is repeated 3 times per generated job (once per if: condition) since GitHub Actions if: expressions can't share a named value — this is inherent to the platform, not a defect, so no action needed.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 129.1 AIC · ⌖ 14 AIC · ⊞ 8.1K

pelikhan · 2026-10-03T22:34:10Z

@copilot resolve the merge conflicts on this branch.

pelikhan · 2026-10-03T22:42:35Z

@copilot resolve the merge conflicts on this branch.

…en-telemetry # Conflicts: # .github/workflows/smoke-copilot-aoai-entra.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-10-03T22:51:33Z

@copilot resolve the merge conflicts on this branch.

Merged main and resolved the lock-file conflict in merge commit 007e68345e.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

…en-telemetry Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-10-03T23:02:23Z

@copilot resolve the merge conflicts on this branch.

Merged the latest main commit cleanly as 0483900aaf; the branch has no unresolved conflicts.

github-actions · 2026-10-05T23:09:50Z

🎉 This pull request is included in a new release.

Release: v0.91.0

Initial plan

b9b0855

Copilot AI assigned Copilot and pelikhan Oct 3, 2026

Copilot started work on behalf of pelikhan October 3, 2026 19:08 View session

Copilot AI linked an issue Oct 3, 2026 that may be closed by this pull request

[AW Top 10] 04 Fix check_token_telemetry for Kiro, Goose, Cursor #65375

Closed

1 task

Copilot AI and others added 2 commits October 3, 2026 19:20

Make smoke token telemetry check engine-aware

ba6ffa0

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Apply remaining changes

ee20d35

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix check_token_telemetry for Kiro, Goose, Cursor~~ Make smoke token telemetry checks engine-aware Oct 3, 2026

Copilot finished work on behalf of pelikhan October 3, 2026 19:25

Copilot AI requested a review from pelikhan October 3, 2026 19:25

pelikhan marked this pull request as ready for review October 3, 2026 19:44

Copilot AI balanced review requested due to automatic review settings October 3, 2026 19:44

Copilot started reviewing on behalf of pelikhan October 3, 2026 19:45 View session

Copilot AI reviewed Oct 3, 2026

View reviewed changes

Comment thread .github/skills/agentic-workflows/SKILL.md

- `.github/aw/update-agentic-workflow.md`

- `.github/aw/upgrade-agentic-workflows.md`

- `.github/aw/visual-regression.md`

- `.github/aw/work-queue.md`

github-actions Bot reviewed Oct 3, 2026

View reviewed changes

github-actions Bot mentioned this pull request Oct 3, 2026

[aw] Design Decision Gate 🏗️ produced no safe outputs #65363

Closed

github-actions Bot added the needs-review label Oct 3, 2026

github-actions Bot reviewed Oct 3, 2026

View reviewed changes

github-actions Bot approved these changes Oct 3, 2026

View reviewed changes

Copilot started work on behalf of pelikhan October 3, 2026 22:34 View session

Merge remote-tracking branch 'origin/main' into copilot/fix-check-tok…

007e683

…en-telemetry # Conflicts: # .github/workflows/smoke-copilot-aoai-entra.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan October 3, 2026 22:52

Copilot started work on behalf of pelikhan October 3, 2026 22:52 View session

Copilot AI and others added 2 commits October 3, 2026 22:58

Plan latest main merge

4843331

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into copilot/fix-check-tok…

0483900

…en-telemetry Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan October 3, 2026 23:02

pelikhan merged commit b1324bb into main Oct 3, 2026
1 check passed

pelikhan deleted the copilot/fix-check-token-telemetry branch October 3, 2026 23:11

timmo001 mentioned this pull request Oct 4, 2026

Update skill: agentic-workflows timmo001/skills#234

Merged

Conversation

Copilot AI commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

github-actions Bot commented Oct 3, 2026

Comment Memory

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Verdict

Comments that could not be inline-anchored

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Key Themes

Positive Highlights

Uh oh!

github-actions Bot Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

pelikhan commented Oct 3, 2026

Uh oh!

pelikhan commented Oct 3, 2026

Uh oh!

Copilot AI commented Oct 3, 2026

Uh oh!

Copilot AI commented Oct 3, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Oct 3, 2026 •

edited

Loading

github-actions Bot commented Oct 3, 2026 •

edited

Loading

github-actions Bot commented Oct 3, 2026 •

edited

Loading

github-actions Bot commented Oct 3, 2026 •

edited

Loading

github-actions Bot commented Oct 3, 2026 •

edited

Loading

github-actions Bot commented Oct 3, 2026 •

edited

Loading