Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/src/content/docs/reference/audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,15 @@ The Friction section attributes execution-friction costs by driver and source. I

The Gateway Steering Events section reports `token_steering` and `timeout_steering` warnings emitted when a run approaches its AI Credits or time limit. JSON output includes each event's type, message, and timestamp when available.

The Work Queue section reports the activation-time queue snapshot, the assigned worker's work and claim IDs, the agent's finish intent, and timestamped coordinator operations from workflow job logs. In JSON, `work_queue.snapshot.transactions` contains the queue facts captured at activation, while `work_queue.operations` contains this run's publication and reconciliation diagnostics with their source files. A `finish_intent` is a request, not evidence that completion was verified. The field is omitted when no coordinator evidence is available.

`audit` downloads this evidence by default. For `logs`, which defaults to compact usage downloads, select `work-queue` (activation and agent artifacts plus workflow logs) or `all`. Both commands include `work_queue` in their JSON reports, per run for `logs`, and preserve it in cached summaries:

```bash
gh aw logs 1234567890 --artifacts work-queue --json
gh aw audit 1234567890 --artifacts work-queue --json
```

The Observability Insights section includes `skill_activations` when skill-invocation evidence is found. Each entry reports the skill name, `status` (`invoked`), the detection `source` (`agent_output` or `log_parse`), and provenance fields in JSON output. This makes it possible to distinguish skills that were merely restored or installed from skills that were actually invoked during the run.

The Graders section is present when the run recorded deterministic grader results (`graders` declared in the workflow frontmatter). The `graders` object in JSON output lists each grader (`id`, `name`, `status`, `value`, `unit`, `passed`, and, when declared in the grader manifest, `direction` and `threshold`) plus aggregate counts: `total`, `passed`, `failed`, `error_count`, and `unavailable_count`. Grader results are read from the compact `usage` artifact (mirrored there by the conclusion job), the unified `agent` artifact, or the `agent-output-fallback` artifact, so they are available even when `--artifacts usage` narrows the download. The same `graders` object is included per run in `gh aw logs --json` output.
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/setup/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -598,7 +598,7 @@ gh aw audit 1234567890 --runtime cloud-hypervisor # Skip run unless sandb

The `--repo` flag accepts `owner/repo` format and is required when passing a bare numeric run ID without a full URL, allowing the command to locate the correct repository.

The `--artifacts` flag selects which artifact sets to download (default: `all`). Valid sets include `activation`, `agent`, `all`, `detection`, `evals`, `experiment`, `firewall`, `github-api`, `graders`, `info`, `mcp`, and `usage`. Use `all` to download the full artifact set. Unlike `gh aw logs`, which defaults to `usage`, `audit` defaults to `all` for comprehensive analysis. The `--experiment` flag filters to runs that include the named experiment; `--variant` further restricts to a specific variant value and requires `--experiment` to be set. The `--output/-o` flag overrides the output directory.
The `--artifacts` flag selects which artifact sets to download (default: `all`). Valid sets include `activation`, `agent`, `all`, `detection`, `evals`, `experiment`, `firewall`, `github-api`, `graders`, `info`, `mcp`, `usage`, and `work-queue`. The `work-queue` set downloads queue snapshots, worker finish intents, and workflow logs for coordinator operation reporting. Use `all` to download the full artifact set. Unlike `gh aw logs`, which defaults to `usage`, `audit` defaults to `all` for comprehensive analysis. The `--experiment` flag filters to runs that include the named experiment; `--variant` further restricts to a specific variant value and requires `--experiment` to be set. The `--output/-o` flag overrides the output directory.

Logs are saved to `.github/aw/logs/run-{id}/` with filenames indicating the extraction level. Pre-agent failures (integrity filtering, missing secrets, binary install) surface the actual error in `failure_analysis.error_summary`. Invalid run IDs return a human-readable error.

Expand Down
1 change: 1 addition & 0 deletions pkg/cli/audit.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ type auditAnalysisResults struct {
tokenUsageSummary *TokenUsageSummary
workingSet *WorkingSetMetrics
ledger *LedgerActivity
dispatchCoordinator *DispatchCoordinatorReport
friction *FrictionCostSummary
redactedDomainsAnalysis *RedactedDomainsAnalysis
rateLimitUsage *GitHubRateLimitUsage
Expand Down
5 changes: 5 additions & 0 deletions pkg/cli/audit_analysis_fanout.go
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,11 @@ func launchFirewallAnalysis(g *errgroup.Group, gctx context.Context, results *au

// launchSupplementalAuditAnalyses exclusively writes redactedDomainsAnalysis, rateLimitUsage, artifacts, and safeItemsCount.
func launchSupplementalAuditAnalyses(g *errgroup.Group, gctx context.Context, results *auditAnalysisResults, runOutputDir string, verbose bool) {
runAuditAnalysis(g, gctx, true, "extractDispatchCoordinatorReport", "Failed to extract dispatch coordinator operations", func(v *DispatchCoordinatorReport) {
results.dispatchCoordinator = v
}, func() (*DispatchCoordinatorReport, error) {
return extractDispatchCoordinatorReport(runOutputDir)
})
runAuditAnalysis(g, gctx, verbose, "analyzeRedactedDomains", "Failed to analyze redacted domains", func(v *RedactedDomainsAnalysis) {
results.redactedDomainsAnalysis = v
}, func() (*RedactedDomainsAnalysis, error) {
Expand Down
8 changes: 7 additions & 1 deletion pkg/cli/audit_cache.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import (

type auditCacheSource string

const auditSchemaVersion = 2
const auditSchemaVersion = 4

const (
auditCacheSourceFull auditCacheSource = "full"
Expand Down Expand Up @@ -99,6 +99,9 @@ func writeLogsAuditFile(processedRun ProcessedRun, processedRuns []ProcessedRun,
auditData, _ = buildLocalAuditData(processedRun, metrics, processedRun.MCPToolUsage)
auditData.CacheSource = auditCacheSourceLogs
}
if processedRun.DispatchCoordinator != nil {
auditData.DispatchCoordinator = processedRun.DispatchCoordinator
}
hydratedProcessedRuns := hydrateProcessedRunsWithCachedAudit(processedRuns)
auditData.Comparison = buildAuditComparisonForProcessedRuns(hydrateProcessedRunWithCachedAudit(processedRun), hydratedProcessedRuns)
if err := writeAuditData(runOutputDir, auditData); err != nil {
Expand Down Expand Up @@ -141,5 +144,8 @@ func hydrateProcessedRunWithCachedAudit(processedRun ProcessedRun) ProcessedRun
if len(processedRun.GatewaySteeringEvents) == 0 {
processedRun.GatewaySteeringEvents = audit.GatewaySteeringEvents
}
if processedRun.DispatchCoordinator == nil {
processedRun.DispatchCoordinator = audit.DispatchCoordinator
}
return processedRun
}
6 changes: 6 additions & 0 deletions pkg/cli/audit_render_output.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ func renderAuditReport(ctx context.Context, processedRun ProcessedRun, metrics L
if auditData.Ledger == nil {
auditData.Ledger = processedRun.Ledger
}
if mergeDispatchCoordinatorReport(&auditData.DispatchCoordinator, processedRun.DispatchCoordinator) && !opts.NoBaseline {
Comment thread
pelikhan marked this conversation as resolved.
Resolved
if err := writeAuditData(runOutputDir, auditData); err != nil {
return err
}
}
if opts.Group {
parseAuditLogsIfRequested(runID, runOutputDir, opts)
return nil
Expand All @@ -64,6 +69,7 @@ func buildRenderedAuditDataFromCache(ctx context.Context, processedRun Processed
if auditData.Ledger == nil {
auditData.Ledger = processedRun.Ledger
}
mergeDispatchCoordinatorReport(&auditData.DispatchCoordinator, processedRun.DispatchCoordinator)
createdItems := resolveCreatedItems(runOutputDir, processedRun.SafeOutputs)
addAuditOutcomeSummary(ctx, &auditData, createdItems)
auditData.Comparison = buildRenderedAuditComparison(ctx, processedRun, createdItems, runOutputDir, opts)
Expand Down
84 changes: 43 additions & 41 deletions pkg/cli/audit_report.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,47 +30,48 @@ var ghaLogTimestampPrefixPattern = regexp.MustCompile(`^\d{4}-\d{2}-\d{2}T`)

// AuditData represents the complete structured audit data for a workflow run
type AuditData struct {
SchemaVersion int `json:"schema_version"`
CacheSource auditCacheSource `json:"cache_source,omitempty"`
Overview OverviewData `json:"overview"`
Comparison *AuditComparisonData `json:"comparison,omitempty"`
TaskDomain *TaskDomainInfo `json:"task_domain,omitempty"`
BehaviorFingerprint *BehaviorFingerprint `json:"behavior_fingerprint,omitempty"`
AgenticAssessments []AgenticAssessment `json:"agentic_assessments,omitempty"`
Metrics MetricsData `json:"metrics"`
KeyFindings []AuditFinding `json:"key_findings,omitempty"`
Recommendations []Recommendation `json:"recommendations,omitempty"`
ObservabilityInsights []ObservabilityInsight `json:"observability_insights,omitempty"`
PerformanceMetrics *PerformanceMetrics `json:"performance_metrics,omitempty"`
EngineConfig *AuditEngineConfig `json:"engine_config,omitempty"`
PromptAnalysis *PromptAnalysis `json:"prompt_analysis,omitempty"`
SessionAnalysis *SessionAnalysis `json:"session_analysis,omitempty"`
SafeOutputSummary *SafeOutputSummary `json:"safe_output_summary,omitempty"`
MCPServerHealth *MCPServerHealth `json:"mcp_server_health,omitempty"`
Jobs []JobData `json:"jobs,omitempty"`
DownloadedFiles []FileInfo `json:"downloaded_files"`
MissingTools []MissingToolReport `json:"missing_tools,omitempty"`
MissingData []MissingDataReport `json:"missing_data,omitempty"`
Noops []NoopReport `json:"noops,omitempty"`
MCPFailures []MCPFailureReport `json:"mcp_failures,omitempty"`
SkillActivations []SkillActivation `json:"skill_activations,omitempty"`
GatewaySteeringEvents []GatewaySteeringEvent `json:"gateway_steering_events,omitempty"`
FirewallTokenUsage *TokenUsageSummary `json:"firewall_token_usage,omitempty"`
GitHubRateLimitUsage *GitHubRateLimitUsage `json:"github_rate_limit_usage,omitempty"`
FirewallAnalysis *FirewallAnalysis `json:"firewall_analysis,omitempty"`
PolicyAnalysis *PolicyAnalysis `json:"policy_analysis,omitempty"`
RedactedDomainsAnalysis *RedactedDomainsAnalysis `json:"redacted_domains_analysis,omitempty"`
Errors []ValidationIssue `json:"errors,omitempty"`
Warnings []ValidationIssue `json:"warnings,omitempty"`
ToolUsage []ToolUsageInfo `json:"tool_usage,omitempty"`
MCPToolUsage *MCPToolUsageData `json:"mcp_tool_usage,omitempty"`
CreatedItems []CreatedItemReport `json:"created_items,omitempty"`
Outcomes []OutcomeReport `json:"outcomes,omitempty"`
OutcomeSummary *OutcomeSummary `json:"outcome_summary,omitempty"`
Experiments *ExperimentData `json:"experiments,omitempty"`
Graders *GradersData `json:"graders,omitempty"`
Ledger *LedgerActivity `json:"ledger,omitempty"`
Friction *FrictionCostSummary `json:"friction,omitempty"`
SchemaVersion int `json:"schema_version"`
CacheSource auditCacheSource `json:"cache_source,omitempty"`
Overview OverviewData `json:"overview"`
Comparison *AuditComparisonData `json:"comparison,omitempty"`
TaskDomain *TaskDomainInfo `json:"task_domain,omitempty"`
BehaviorFingerprint *BehaviorFingerprint `json:"behavior_fingerprint,omitempty"`
AgenticAssessments []AgenticAssessment `json:"agentic_assessments,omitempty"`
Metrics MetricsData `json:"metrics"`
KeyFindings []AuditFinding `json:"key_findings,omitempty"`
Recommendations []Recommendation `json:"recommendations,omitempty"`
ObservabilityInsights []ObservabilityInsight `json:"observability_insights,omitempty"`
PerformanceMetrics *PerformanceMetrics `json:"performance_metrics,omitempty"`
EngineConfig *AuditEngineConfig `json:"engine_config,omitempty"`
PromptAnalysis *PromptAnalysis `json:"prompt_analysis,omitempty"`
SessionAnalysis *SessionAnalysis `json:"session_analysis,omitempty"`
SafeOutputSummary *SafeOutputSummary `json:"safe_output_summary,omitempty"`
MCPServerHealth *MCPServerHealth `json:"mcp_server_health,omitempty"`
Jobs []JobData `json:"jobs,omitempty"`
DownloadedFiles []FileInfo `json:"downloaded_files"`
MissingTools []MissingToolReport `json:"missing_tools,omitempty"`
MissingData []MissingDataReport `json:"missing_data,omitempty"`
Noops []NoopReport `json:"noops,omitempty"`
MCPFailures []MCPFailureReport `json:"mcp_failures,omitempty"`
SkillActivations []SkillActivation `json:"skill_activations,omitempty"`
GatewaySteeringEvents []GatewaySteeringEvent `json:"gateway_steering_events,omitempty"`
FirewallTokenUsage *TokenUsageSummary `json:"firewall_token_usage,omitempty"`
GitHubRateLimitUsage *GitHubRateLimitUsage `json:"github_rate_limit_usage,omitempty"`
FirewallAnalysis *FirewallAnalysis `json:"firewall_analysis,omitempty"`
PolicyAnalysis *PolicyAnalysis `json:"policy_analysis,omitempty"`
RedactedDomainsAnalysis *RedactedDomainsAnalysis `json:"redacted_domains_analysis,omitempty"`
Errors []ValidationIssue `json:"errors,omitempty"`
Warnings []ValidationIssue `json:"warnings,omitempty"`
ToolUsage []ToolUsageInfo `json:"tool_usage,omitempty"`
MCPToolUsage *MCPToolUsageData `json:"mcp_tool_usage,omitempty"`
CreatedItems []CreatedItemReport `json:"created_items,omitempty"`
Outcomes []OutcomeReport `json:"outcomes,omitempty"`
OutcomeSummary *OutcomeSummary `json:"outcome_summary,omitempty"`
Experiments *ExperimentData `json:"experiments,omitempty"`
Graders *GradersData `json:"graders,omitempty"`
Ledger *LedgerActivity `json:"ledger,omitempty"`
DispatchCoordinator *DispatchCoordinatorReport `json:"work_queue,omitempty"`
Friction *FrictionCostSummary `json:"friction,omitempty"`
}

// AuditFinding represents a key insight discovered during audit
Expand Down Expand Up @@ -560,6 +561,7 @@ func assembleAuditData(inputs auditDataInputs) AuditData {
Metrics: metricsData,
Friction: inputs.processedRun.Friction,
Ledger: inputs.processedRun.Ledger,
DispatchCoordinator: inputs.processedRun.DispatchCoordinator,
KeyFindings: inputs.findings,
Recommendations: inputs.recommendations,
ObservabilityInsights: inputs.observabilityInsights,
Expand Down
2 changes: 2 additions & 0 deletions pkg/cli/audit_report_render.go
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,7 @@ func renderConsoleWarnings(warnings []ValidationIssue) {
}

func renderConsoleOperationalSections(data AuditData) {
renderConsoleDispatchCoordinator(data.DispatchCoordinator)
renderConsoleSkillActivations(data.SkillActivations)
renderConsoleGatewaySteeringEvents(data.GatewaySteeringEvents)
renderConsoleMissingTools(data.MissingTools)
Expand All @@ -341,6 +342,7 @@ func renderConsoleOperationalSections(data AuditData) {
if data.FirewallAnalysis != nil && data.FirewallAnalysis.TotalRequests > 0 {
renderCompactFirewall(data.FirewallAnalysis)
}

}

// renderConsoleFriction prints the per-driver breakdown and the individual
Expand Down
8 changes: 8 additions & 0 deletions pkg/cli/audit_run_pipeline.go
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,12 @@ func renderCachedAuditIfAvailable(ctx context.Context, cfg auditRunConfig) (done
if !ok {
return false, false, nil
}
if dispatchCoordinatorArtifactsRequested(cfg.artifactFilter) && len(findMissingFilterEntries(cfg.artifactFilter, cfg.outputDir)) > 0 {
Comment thread
pelikhan marked this conversation as resolved.
Resolved
return false, false, nil
}
if dispatchCoordinatorLogsMissing(cfg.artifactFilter, cfg.outputDir) {
return false, false, nil
}
auditLog.Printf("Using cached run summary for run %d (processed at %s)", cfg.runID, summary.ProcessedAt.Format(time.RFC3339))
if cfg.verbose {
fmt.Fprintln(os.Stderr, console.FormatInfoMessage(fmt.Sprintf("Using cached run summary for run %d (processed at %s)", cfg.runID, summary.ProcessedAt.Format(time.RFC3339))))
Expand Down Expand Up @@ -230,6 +236,7 @@ func auditNeedsDetectionArtifact(cfg auditRunConfig, summary *RunSummary) bool {
}

func processedRunFromSummary(summary *RunSummary, runOutputDir string) ProcessedRun {
backfillDispatchCoordinatorReport(&summary.DispatchCoordinator, runOutputDir)
if summary.Ledger == nil {
if activity, err := loadUsageActivitySummary(runOutputDir); err == nil && activity != nil {
summary.Ledger = activity.Ledger
Expand Down Expand Up @@ -261,6 +268,7 @@ func processedRunFromSummary(summary *RunSummary, runOutputDir string) Processed
SafeOutputs: summary.SafeOutputs,
WorkingSet: summary.WorkingSet,
Ledger: summary.Ledger,
DispatchCoordinator: summary.DispatchCoordinator,
Friction: summary.Friction,
GitHubRateLimitUsage: summary.GitHubRateLimitUsage,
JobDetails: summary.JobDetails,
Expand Down
2 changes: 2 additions & 0 deletions pkg/cli/audit_summary_build.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ func buildProcessedAuditRun(run WorkflowRun, results auditAnalysisResults) Proce
TokenUsage: results.tokenUsageSummary,
WorkingSet: results.workingSet,
Ledger: results.ledger,
DispatchCoordinator: results.dispatchCoordinator,
Friction: results.friction,
GitHubRateLimitUsage: results.rateLimitUsage,
JobDetails: results.jobDetails,
Expand Down Expand Up @@ -83,6 +84,7 @@ func buildAuditRunSummary(run WorkflowRun, processedRun ProcessedRun, results au
GatewaySteeringEvents: results.gatewaySteeringEvents,
WorkingSet: results.workingSet,
Ledger: results.ledger,
DispatchCoordinator: results.dispatchCoordinator,
Friction: results.friction,
GitHubRateLimitUsage: results.rateLimitUsage,
JobDetails: results.jobDetails,
Expand Down
Loading
Loading