Skip to content

Commit c353937

Browse files
Copilotpelikhangithub-actions[bot]
authored
Report gateway steering events in audit output (#62943)
* Report gateway steering events in audits Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Add ADR for gateway steering audit reporting * Fix gateway steering cache propagation Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 parent 7f9138d commit c353937

23 files changed

Lines changed: 380 additions & 40 deletions

‎.changeset/minor-audit-gateway-steering.md‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# ADR-62943: Report gateway steering events in audit output
2+
3+
**Date**: 2026-09-23
4+
**Status**: Draft
5+
**Deciders**: gh-aw maintainers
6+
7+
---
8+
9+
### Context
10+
11+
The `gh aw audit` command currently counts gateway steering events but does not preserve the event details that explain whether a run is nearing AI-credit exhaustion or its time limit. The PR description and diff show downstream users need those structured warnings in both console and JSON audit output, and cached summaries must carry the same data so repeated audits remain complete. The change spans parsing firewall gateway logs, extending in-memory and cached audit models, rendering a new report section, and updating published schemas and documentation. The decision is whether audit should keep reporting only aggregate steering counts or promote gateway steering warnings into a first-class structured report field.
12+
13+
### Decision
14+
15+
We will expose gateway steering warnings as structured `gateway_steering_events` in audit output, cache them in run summaries, and render them in the console report alongside other operational sections. We will extract both `token_steering` and `timeout_steering` events from gateway log entries, preserving each event's type, message, and timestamp when available. We chose this approach because the PR evidence shows aggregate counts alone obscure the operational reason for steering, while a structured field keeps JSON, cached data, console output, schemas, and documentation aligned.
16+
17+
### Alternatives Considered
18+
19+
#### Alternative 1: Keep reporting only aggregate steering counts
20+
21+
This matches the prior implementation and is the lowest-effort option because it reuses the existing token-usage summary without adding new report fields. It was not chosen because the PR explicitly addresses the gap that counts do not tell operators whether warnings came from AI-credit pressure or time pressure, which makes audit output less actionable.
22+
23+
#### Alternative 2: Surface steering details only in raw logs or documentation
24+
25+
This was a realistic option because the gateway events already exist in firewall log files, and users could inspect those logs outside the audit report. It was not chosen because the diff adds tests, report wiring, cache hydration, and schema changes intended to make steering details part of the supported audit contract rather than an implementation detail hidden in downloaded artifacts.
26+
27+
### Consequences
28+
29+
#### Positive
30+
- Audit consumers can distinguish AI-credit warnings from time-limit warnings directly from console and JSON output.
31+
- Cached summaries and regenerated audit reports remain complete because the new field is stored in `RunSummary` and `ProcessedRun`.
32+
- Published schemas and documentation explicitly describe the new contract, reducing ambiguity for tooling that parses audit output.
33+
34+
#### Negative
35+
- The audit data model and rendering pipeline become more complex because a new structured event type must be threaded through parsing, caching, serialization, and presentation layers.
36+
- The audit cache schema version must be bumped, which invalidates older cached reports and forces regeneration.
37+
- Tests and schemas now need ongoing maintenance whenever gateway steering event semantics change.
38+
39+
#### Neutral
40+
- Existing aggregate token-usage analysis remains in place; this change augments it with event detail rather than replacing it.
41+
- The new report section appears only when gateway steering events are present, so runs without such warnings keep their current output shape apart from the expanded schema.
42+
43+
---
44+
45+
*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.*

‎docs/src/content/docs/reference/audit.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,9 @@ gh aw audit 12345 12346 --json # JSON for CI integration
6868
gh aw audit 12345 12346 --repo owner/repo # Specify repository
6969
```
7070

71-
**Single-run report sections** (rendered in Markdown or JSON): Overview, Comparison, Task/Domain, Behavior Fingerprint, Agentic Assessments, Metrics, Key Findings, Recommendations, Observability Insights, Performance Metrics, Engine Config, Prompt Analysis, Session Analysis, Safe Output Summary, MCP Server Health, Jobs, Downloaded Files, Missing Tools, Missing Data, Noops, MCP Failures, Firewall Analysis, Policy Analysis, Redacted Domains, Errors, Warnings, Tool Usage, MCP Tool Usage, Created Items, Graders.
71+
**Single-run report sections** (rendered in Markdown or JSON): Overview, Comparison, Task/Domain, Behavior Fingerprint, Agentic Assessments, Metrics, Key Findings, Recommendations, Observability Insights, Performance Metrics, Engine Config, Prompt Analysis, Session Analysis, Safe Output Summary, MCP Server Health, Jobs, Downloaded Files, Missing Tools, Missing Data, Noops, MCP Failures, Gateway Steering Events, Firewall Analysis, Policy Analysis, Redacted Domains, Errors, Warnings, Tool Usage, MCP Tool Usage, Created Items, Graders.
72+
73+
The Gateway Steering Events section reports `token_steering` and `timeout_steering` warnings emitted when a run approaches its AI Credits or time limit. JSON output includes each event's type, message, and timestamp when available.
7274

7375
The Observability Insights section includes `skill_activations` when skill-invocation evidence is found. Each entry reports the skill name, `status` (`invoked`), the detection `source` (`agent_output` or `log_parse`), and provenance fields in JSON output. This makes it possible to distinguish skills that were merely restored or installed from skills that were actually invoked during the run.
7476

‎pkg/cli/audit.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ type auditAnalysisResults struct {
6767
noops []NoopReport
6868
mcpFailures []MCPFailureReport
6969
skillActivations []SkillActivation
70+
gatewaySteeringEvents []GatewaySteeringEvent
7071
accessAnalysis *DomainAnalysis
7172
firewallAnalysis *FirewallAnalysis
7273
policyAnalysis *PolicyAnalysis

‎pkg/cli/audit_agent_output_test.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -361,6 +361,9 @@ func TestAuditDataJSONStructure(t *testing.T) {
361361
MCPFailures: []MCPFailureReport{
362362
{ServerName: "test-server", Status: "failed"},
363363
},
364+
GatewaySteeringEvents: []GatewaySteeringEvent{
365+
{Type: tokenSteeringEventName, Message: "[AWF TOKEN WARNING] You are running out of AI Credits."},
366+
},
364367
JobDetails: []JobInfoWithDuration{
365368
{JobInfo: JobInfo{Name: "test", Conclusion: "failure"}},
366369
},
@@ -388,6 +391,7 @@ func TestAuditDataJSONStructure(t *testing.T) {
388391
"downloaded_files",
389392
"missing_tools",
390393
"mcp_failures",
394+
"gateway_steering_events",
391395
"tool_usage",
392396
}
393397

‎pkg/cli/audit_analysis_fanout.go‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -132,7 +132,7 @@ func launchJobDetailsAnalysis(g *errgroup.Group, gctx context.Context, results *
132132
})
133133
}
134134

135-
// launchFirewallAuditAnalyses exclusively writes policyAnalysis, mcpToolUsage, and tokenUsageSummary.
135+
// launchFirewallAuditAnalyses exclusively writes policyAnalysis, mcpToolUsage, tokenUsageSummary, and gatewaySteeringEvents.
136136
func launchFirewallAuditAnalyses(g *errgroup.Group, gctx context.Context, results *auditAnalysisResults, runOutputDir string, verbose bool) {
137137
launchFirewallAnalysis(g, gctx, results, runOutputDir, verbose)
138138
runAuditAnalysis(g, gctx, verbose, "analyzeFirewallPolicy", "Failed to analyze firewall policy", func(v *PolicyAnalysis) {
@@ -150,6 +150,11 @@ func launchFirewallAuditAnalyses(g *errgroup.Group, gctx context.Context, result
150150
}, func() (*TokenUsageSummary, error) {
151151
return analyzeTokenUsage(runOutputDir, verbose)
152152
})
153+
runAuditAnalysis(g, gctx, verbose, "extractGatewaySteeringEvents", "Failed to extract gateway steering events", func(v []GatewaySteeringEvent) {
154+
results.gatewaySteeringEvents = v
155+
}, func() ([]GatewaySteeringEvent, error) {
156+
return extractGatewaySteeringEvents(runOutputDir)
157+
})
153158
}
154159

155160
// launchFirewallAnalysis exclusively writes results.firewallAnalysis.

‎pkg/cli/audit_cache.go‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ import (
1313

1414
type auditCacheSource string
1515

16-
const auditSchemaVersion = 1
16+
const auditSchemaVersion = 2
1717

1818
const (
1919
auditCacheSourceFull auditCacheSource = "full"
@@ -135,5 +135,8 @@ func hydrateProcessedRunWithCachedAudit(processedRun ProcessedRun) ProcessedRun
135135
if len(processedRun.MCPFailures) == 0 {
136136
processedRun.MCPFailures = audit.MCPFailures
137137
}
138+
if len(processedRun.GatewaySteeringEvents) == 0 {
139+
processedRun.GatewaySteeringEvents = audit.GatewaySteeringEvents
140+
}
138141
return processedRun
139142
}

‎pkg/cli/audit_render_output_test.go‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,21 @@ func TestRenderConsoleTokenUsageWarnings(t *testing.T) {
6868
assert.Contains(t, output, "fallback accounting was used")
6969
}
7070

71+
func TestRenderConsoleGatewaySteeringEvents(t *testing.T) {
72+
output := testutil.CaptureStderr(t, func() {
73+
renderConsoleGatewaySteeringEvents([]GatewaySteeringEvent{{
74+
Type: tokenSteeringEventName,
75+
Message: "[AWF TOKEN WARNING] You are running out of AI Credits.",
76+
Timestamp: "2026-09-23T12:00:00Z",
77+
}})
78+
})
79+
80+
assert.Contains(t, output, "gateway_steering_events:")
81+
assert.Contains(t, output, tokenSteeringEventName)
82+
assert.Contains(t, output, "running out of AI Credits")
83+
assert.Contains(t, output, "2026-09-23T12:00:00Z")
84+
}
85+
7186
func TestRenderAuditCompletion(t *testing.T) {
7287
outputDir := t.TempDir()
7388

‎pkg/cli/audit_report.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@ type AuditData struct {
5454
Noops []NoopReport `json:"noops,omitempty"`
5555
MCPFailures []MCPFailureReport `json:"mcp_failures,omitempty"`
5656
SkillActivations []SkillActivation `json:"skill_activations,omitempty"`
57+
GatewaySteeringEvents []GatewaySteeringEvent `json:"gateway_steering_events,omitempty"`
5758
FirewallTokenUsage *TokenUsageSummary `json:"firewall_token_usage,omitempty"`
5859
GitHubRateLimitUsage *GitHubRateLimitUsage `json:"github_rate_limit_usage,omitempty"`
5960
FirewallAnalysis *FirewallAnalysis `json:"firewall_analysis,omitempty"`
@@ -537,6 +538,7 @@ func assembleAuditData(inputs auditDataInputs) AuditData {
537538
Noops: inputs.processedRun.Noops,
538539
MCPFailures: inputs.processedRun.MCPFailures,
539540
SkillActivations: inputs.processedRun.SkillActivations,
541+
GatewaySteeringEvents: inputs.processedRun.GatewaySteeringEvents,
540542
FirewallTokenUsage: inputs.processedRun.TokenUsage,
541543
GitHubRateLimitUsage: inputs.processedRun.GitHubRateLimitUsage,
542544
FirewallAnalysis: inputs.processedRun.FirewallAnalysis,

‎pkg/cli/audit_report_render.go‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -312,6 +312,7 @@ func renderConsoleWarnings(warnings []ValidationIssue) {
312312

313313
func renderConsoleOperationalSections(data AuditData) {
314314
renderConsoleSkillActivations(data.SkillActivations)
315+
renderConsoleGatewaySteeringEvents(data.GatewaySteeringEvents)
315316
renderConsoleMissingTools(data.MissingTools)
316317
renderConsoleMCPFailures(data.MCPFailures)
317318
renderCompactMCPHealth(data.MCPServerHealth)
@@ -324,6 +325,20 @@ func renderConsoleOperationalSections(data AuditData) {
324325
}
325326
}
326327

328+
func renderConsoleGatewaySteeringEvents(events []GatewaySteeringEvent) {
329+
if len(events) == 0 {
330+
return
331+
}
332+
fmt.Fprintln(os.Stderr, " gateway_steering_events:")
333+
for _, event := range events {
334+
line := fmt.Sprintf(" %s: %s", event.Type, event.Message)
335+
if event.Timestamp != "" {
336+
line += " (" + event.Timestamp + ")"
337+
}
338+
fmt.Fprintln(os.Stderr, line)
339+
}
340+
}
341+
327342
func renderConsoleSkillActivations(activations []SkillActivation) {
328343
if len(activations) == 0 {
329344
return

0 commit comments

Comments
 (0)