Skip to content

Make smoke token telemetry checks engine-aware #65403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 6 commits into from
Oct 3, 2026
Merged

Make smoke token telemetry checks engine-aware #65403

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b9b0855
Initial plan
Copilot Oct 3, 2026
ba6ffa0
Make smoke token telemetry check engine-aware
Copilot Oct 3, 2026
ee20d35
Apply remaining changes
Copilot Oct 3, 2026
007e683
Merge remote-tracking branch 'origin/main' into copilot/fix-check-tok…
Copilot Oct 3, 2026
4843331
Plan latest main merge
Copilot Oct 3, 2026
0483900
Merge remote-tracking branch 'origin/main' into copilot/fix-check-tok…
Copilot Oct 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .github/skills/agentic-workflows/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,10 +87,10 @@ Load these files from `github/gh-aw` (they are not available locally).
- `.github/aw/update-agentic-workflow.md`
- `.github/aw/upgrade-agentic-workflows.md`
- `.github/aw/visual-regression.md`
- `.github/aw/work-queue.md`
- `.github/aw/workflow-constraints.md`
- `.github/aw/workflow-editing.md`
- `.github/aw/workflow-patterns.md`

@github-actions github-actions Bot Oct 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/grill-with-docs] This skills-ordering tweak (reordering work-queue.md and adding it to the "After loading" bullet list) is unrelated to the PR's stated purpose of making smoke token telemetry checks engine-aware.

💡 Suggested approach

Unrelated doc/skill changes bundled into a focused bug-fix PR make the diff harder to review and the git history harder to bisect later. If this was an unintentional rebase artifact, drop it from this PR; if intentional, split it into its own PR so the token-telemetry fix stays a clean, reviewable unit.

@copilot please address this.

- `.github/aw/work-queue.md`

After loading the matching workflow prompt or skill, follow it directly:
- Design workflows from scratch via interview: `.github/aw/designer.md`
Expand All @@ -106,7 +106,6 @@ After loading the matching workflow prompt or skill, follow it directly:
- Render compact markdown charts: `.github/aw/asciicharts.md`
- Map CLI commands to MCP usage: `.github/aw/cli-commands.md`
- Choose workflow architecture and patterns: `.github/aw/patterns.md`
- Orchestrate durable work with a work queue (dispatcher/worker roles or queue inspection): `.github/aw/work-queue.md`
- Optimize token usage and cost: `.github/aw/token-optimization.md`
- Design long-running multi-agent research workflows: `.github/aw/multi-agent-research.md`
- Add skills or agent plugins requested by the user (`skills:` / `plugins:` frontmatter, never on-the-fly installs): `.github/aw/skills.md`
Expand Down
16 changes: 11 additions & 5 deletions .github/workflows/shared/token-telemetry-check.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,20 @@
jobs:
check_token_telemetry:
runs-on: ubuntu-latest
needs: [agent]
needs: [agent, activation]
if: needs.agent.result == 'success'
permissions:
contents: read
steps:
- name: Download agent artifact
id: download-agent
continue-on-error: true
uses: actions/download-artifact@v8.0.1
with:
name: agent
path: /tmp/gh-aw/

- name: Assert token_usage.jsonl is non-empty
if: steps.download-agent.outcome == 'success'
if: steps.download-agent.outcome == 'success' && !contains(fromJSON('["kiro","goose","cursor"]'), needs.activation.outputs.engine_id)

@github-actions github-actions Bot Oct 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] The set of engines without proxy token telemetry (["kiro","goose","cursor"]) is duplicated as a raw string literal 3 times in this file, with no single source of truth. If a new engine lacking proxy telemetry is added, or kiro/goose/cursor later gain telemetry, every occurrence must be updated in lockstep and the compiled lock files regenerated — easy to miss.

💡 Suggested approach

The codebase already has an EngineCapabilities struct (pkg/workflow/agentic_engine.go) and EnginesWithCapability(predicate) registry helper used to build "registry-driven lists of supported engines" elsewhere. Consider adding a ProxyTokenTelemetry bool capability field, then have the compiler inject the resulting engine-ID list into this shared workflow template (e.g. via a compiler-rendered {{ ... }} placeholder) instead of hardcoding it in the .md source. This keeps the list correct automatically as engines evolve, matching the stated purpose of EnginesWithCapability.

@copilot please address this.

run: |
# The AWF firewall proxy writes token_usage.jsonl for every LLM API call.
# If all token_usage.jsonl files are missing or empty, the emitter is broken.
Expand Down Expand Up @@ -44,7 +43,7 @@ jobs:
fi

- name: Assert agent_usage.json has non-zero token counts
if: steps.download-agent.outcome == 'success'
if: steps.download-agent.outcome == 'success' && !contains(fromJSON('["kiro","goose","cursor"]'), needs.activation.outputs.engine_id)
run: |
USAGE_FILE="/tmp/gh-aw/agent_usage.json"
if [ ! -f "${USAGE_FILE}" ]; then
Expand All @@ -58,12 +57,19 @@ jobs:
exit 1
fi
echo "OK: agent_usage.json reports ${INPUT_TOKENS} input tokens"

- name: Report engines without proxy token telemetry
if: steps.download-agent.outcome == 'success' && contains(fromJSON('["kiro","goose","cursor"]'), needs.activation.outputs.engine_id)
run: echo "::notice::This engine does not provide proxy token counts; token telemetry assertions are not applicable."
---
<!--
# Token Telemetry Check

This shared workflow adds a `check_token_telemetry` job that runs after the `agent` job
and asserts that AWF firewall proxy token telemetry is functioning correctly.
and asserts that AWF firewall proxy token telemetry is functioning correctly
for engines that provide proxy token counts. Kiro, Goose, and Cursor do not
currently provide these counts, so the job reports that the assertions are
not applicable instead of treating missing proxy records as a regression.

It guards against silent regressions in the `token_usage.jsonl` emitter, which feeds
downstream token-cost and API-consumption audits.
Expand Down
14 changes: 9 additions & 5 deletions .github/workflows/smoke-claude.lock.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 9 additions & 5 deletions .github/workflows/smoke-codex.lock.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading