Skip to content

Fix operational-value grader created-at acquisition and preserve grader result contract in gh aw logs - #60635

Closed
pelikhan with Copilot wants to merge 13 commits into
mainfrom
copilot/fix-null-run-created-at
Closed

pelikhan with Copilot wants to merge 13 commits into
mainfrom
copilot/fix-null-run-created-at

Conversation

Copilot AI commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Operational-value grading was failing at two boundaries: graders received run.createdAt: null and silently graded an invalid request, and gh aw logs --artifacts graders collapsed the rich grader result into a lossy summary (dropping source, implementation, observation, diagnostics, and baseline fields, and rewriting unit from ratio to count). Downstream consumers could not distinguish missing evidence from transport loss.

Created-at acquisition (JS runtime)

  • run_created_at.cjs (new) — shared normalizeRunCreatedAt(): strict ISO-8601 validation before Date.parse, normalization to second-granularity UTC (the format parseTimestamp requires), "" for anything unusable.

  • generate_aw_info.cjs — resolveRunCreatedAt() fails fast when no authenticated client is available, retries transient getWorkflowRun failures with incremental backoff, breaks early on permanent 4xx (except 408/429), and normalizes before core.setOutput("run_created_at", …).

  • trace_graders.cjs — resolveRunCreatedAtForGrading() prefers GH_AW_RUN_CREATED_AT, falls back to Actions run metadata, and names the missing input when neither is usable. The grader loop now enforces "never grade without run metadata": an operational-value grader with no resolved created-at yields an explicit error rather than an invalid request.

    { "id": "operational-value", "status": "error", "unit": "ratio",
      "error": "grader operational-value runtime error: workflow run creation time is unavailable and cannot be read from Actions run metadata (missing an authenticated GitHub client)",
      "diagnostics": { "missingReason": "run created-at acquisition failed" } }
  • operational_value_grader.cjs — buildRunSubject rejects a malformed non-null createdAt instead of passing it through.

Serialization contract (Go)

  • pkg/cli/audit_report_graders.go — GraderResult carries source, implementation (incl. digest), observation, diagnostics, baselineValue, and deltaFromBaseline, read from grader_results.json. The manifest unit is now strictly a fallback, so a ratio metric no longer degrades to count during serialization. Free-form payloads use map[string]any (not json.RawMessage) so the generated schemas render as objects rather than byte arrays.
  • Regenerated schemas/audit.schema.json, schemas/logs.schema.json, schemas/logs-jsonl.schema.json.

Tests, spec, changeset

  • Go round-trip test over a rich operational-value result: numeric zero, null value, ratio unit, maturity fields, baseline fields, diagnostics.
  • JS coverage for normalization/rejection, retry vs. non-retryable failures, metadata fallback, and the grader-loop acquisition-failure branch.
  • graders-specification.md §7 (acquisition requirement) and §8.4 (serialization contract) made normative; patch changeset added.

Note on the root cause

Compiler wiring (GH_AW_INFO_FETCH_RUN_CREATED_AT, the activation run_created_at output, and GH_AW_RUN_CREATED_AT on the graders step) is already correct and the production null could not be reproduced locally, so these changes are defensive: normalize, retry, fall back, and — when all of that fails — report the acquisition failure explicitly instead of emitting an unavailable result with an invalid request. Note that actions: read is scoped to the activation job by design, so the graders-step API fallback may 403; that path is exactly what the new error result surfaces.

  • Fixes #60633

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot Make sure graders information is collected fully in the conclusion job in the usage artifact .json payload

Copilot AI and others added 3 commits September 13, 2026 15:12
…esult contract in logs

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
… schemas

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Comment thread actions/setup/js/generate_aw_info.cjs Outdated
*/
async function resolveRunCreatedAt(core, ctx, githubClient) {
let lastError = "workflow run creation time is unavailable";
for (let attempt = 1; attempt <= RUN_CREATED_AT_ATTEMPTS; attempt++) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot use withRetry

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched to withRetry from error_recovery.cjs (shared RUN_CREATED_AT_RETRY_CONFIG, transient-only shouldRetry), replacing the hand-rolled retry loop. c25ee41

Comment thread actions/setup/js/generate_aw_info.cjs Outdated
const { ERR_CONFIG, ERR_SYSTEM } = require("./error_codes.cjs");

// Retries guard the operational-value grader against transient run-metadata lookup failures.
const RUN_CREATED_AT_ATTEMPTS = 3;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot move run create at logic into own file, add tests
Move constants to constants

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All created-at logic now lives in run_created_at.cjs (normalizeRunCreatedAt, fetchRunCreatedAt, resolveRunCreatedAtForInfo, resolveRunCreatedAtForGrading); generate_aw_info.cjs and trace_graders.cjs just call it. Retry constants moved to constants.cjs, and the tests moved into run_created_at.test.cjs. c25ee41

Copilot AI and others added 3 commits September 13, 2026 15:20
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix null run.createdAt and logs issue in operational-value grading Fix operational-value grader created-at acquisition and preserve grader result contract in gh aw logs Sep 13, 2026
Copilot AI requested a review from pelikhan September 13, 2026 15:26
…e usage summary

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

@copilot Make sure graders information is collected fully in the conclusion job in the usage artifact .json payload

The conclusion job's generate_usage_activity_summary.cjs now embeds the full grader_results.json document (verbatim, including source, value, unit, implementation, observation, diagnostics, and baseline fields) under graders in usage/activity/summary.json, and gh aw logs falls back to that payload when the standalone results file is unavailable. Covered by JS and Go tests, documented in artifacts.md and the graders spec. c25ee41

Copilot AI and others added 2 commits September 13, 2026 15:42
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot there are probably helper functions to resolve files in multiple potential locations, use those helpers

@pelikhan
pelikhan marked this pull request as ready for review September 13, 2026 15:49
Copilot AI balanced review requested due to automatic review settings September 13, 2026 15:49
@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Security scanning failed for Design Decision Gate 🏗️. Review the logs for details.

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Lean already. Ship.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • ab.chatgpt.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "ab.chatgpt.com"

See Network Configuration for more information.

Generated by Ponytail Reviewer for #60635

@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ PR Code Quality Reviewer failed during code quality review.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

🔎 Code quality review by PR Code Quality Reviewer

@pelikhan
pelikhan added this pull request to stack #60646 September 13, 2026 15:54
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

@copilot there are probably helper functions to resolve files in multiple potential locations, use those helpers

There was no shared JS helper for this (only duplicated .find(c => fs.existsSync(c)) idioms), so I added findExistingFiles / findFirstExistingFile to file_helpers.cjs and routed both the grader-results lookup and the existing gateway/rpc log lookup in generate_usage_activity_summary.cjs through them. The Go side already goes through findGraderFile/graderArtifactDirCandidates and loadUsageActivitySummary. ea8d5f3

@github-actions

Copy link
Copy Markdown
Contributor
🏗️ ADR Required — draft added for PR #60635

An ADR was required for this PR because it adds more than 100 lines in business-logic directories (default_business_additions: 267), even though the implementation label is not present.

Evidence reviewed

Gate result

No existing ADR for this decision was found in the PR body or on the branch. I generated and committed a draft ADR:

  • docs/adr/60635-bind-operational-value-grading-to-run-created-at-and-preserve-grader-results.md

Inferred decision

This PR makes one architectural decision: operational-value grading must never proceed without a valid workflow run creation timestamp, and grader results must preserve the full result contract across artifact and CLI boundaries.

Next action

Please review and refine the draft ADR, especially the rationale and trade-offs, before merging this PR.

🏗️ ADR gate enforced by Design Decision Gate 🏗️ · pi · gpt54 · 29.9 AIC · ⊞ 9.7K · ◷
Comment /review to run again

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Null fields and empty diagnostics remain lossy, while timestamp validation and fallback paths have correctness gaps.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Fixes #60633 by improving run-created-at acquisition and preserving operational-value grader data in logs.

Changes:

  • Adds timestamp normalization, retry, fallback, and explicit failure handling.
  • Preserves rich grader fields and embeds results in usage summaries.
  • Adds tests, schemas, documentation, and a patch changeset.
File summaries
File Description
schemas/logs.schema.json Adds rich grader fields.
schemas/logs-jsonl.schema.json Extends JSONL grader schemas.
schemas/audit.schema.json Extends audit grader schema.
pkg/cli/logs_usage_activity.go Parses embedded grader results.
pkg/cli/audit_report_graders.go Expands grader serialization and fallback loading.
pkg/cli/audit_report_graders_test.go Tests rich result round-tripping and fallback.
docs/src/content/docs/specs/graders-specification.md Defines acquisition and serialization requirements.
docs/src/content/docs/reference/artifacts.md Documents embedded grader results.
actions/setup/js/trace_graders.test.cjs Tests acquisition failure handling.
actions/setup/js/trace_graders.cjs Resolves metadata before operational grading.
actions/setup/js/run_created_at.test.cjs Tests timestamp acquisition and normalization.
actions/setup/js/run_created_at.cjs Adds shared timestamp lookup logic.
actions/setup/js/operational_value_grader.test.cjs Tests created-at validation.
actions/setup/js/operational_value_grader.cjs Validates the run creation timestamp.
actions/setup/js/generate_usage_activity_summary.test.cjs Tests grader-result embedding.
actions/setup/js/generate_usage_activity_summary.cjs Embeds grader results in usage summaries.
actions/setup/js/generate_aw_info.test.cjs Tests normalization and retries.
actions/setup/js/generate_aw_info.cjs Uses shared created-at resolution.
actions/setup/js/constants.cjs Adds retry and grader-path constants.
.changeset/operational-value-run-created-at.md Records the patch release.
Review details

Suppressed comments (2)

pkg/cli/audit_report_graders.go:46

  • Explicit null baseline fields are converted to nil pointers and then omitted here. The source operational-value result always carries these fields, and the new specification requires them to survive republishing, so an unavailable baseline must serialize as "baselineValue": null and "deltaFromBaseline": null rather than disappearing.
	BaselineValue     *float64              `json:"baselineValue,omitempty"`
	DeltaFromBaseline *float64              `json:"deltaFromBaseline,omitempty"`

pkg/cli/audit_report_graders.go:44

  • An operational-value evaluator may legitimately emit diagnostics: {} (the updated artifact documentation shows this shape), but an empty map satisfies omitempty and disappears from gh aw logs. This still violates the new requirement to preserve diagnostics and again makes an empty diagnostic payload indistinguishable from transport loss. Represent presence separately or otherwise ensure an explicit empty object survives marshaling.
	Diagnostics       map[string]any        `json:"diagnostics,omitempty"`
  • Files reviewed: 23/23 changed files
  • Comments generated: 5
  • Review effort level: Balanced

Comment thread actions/setup/js/run_created_at.cjs Outdated
Comment on lines +57 to +60
if (!ISO_8601_DATE_TIME.test(trimmed)) return "";
const parsed = Date.parse(trimmed);
if (!Number.isFinite(parsed)) return "";
return new Date(Math.trunc(parsed / 1000) * 1000).toISOString().replace(/\.\d{3}Z$/, "Z");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed timestamp normalization to validate captured calendar/time/offset components before Date.parse, so malformed dates like February 30 are rejected instead of normalized. Added coverage for invalid dates and leap days. ad5eac0

Comment on lines +29 to +32
shouldRetry: error => {
const status = Number(error?.status ?? error?.response?.status);
return (Number.isInteger(status) && status >= 500) || isTransientError(error);
},

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the shared withRetry config to treat status-only HTTP 408 errors as retryable and added a targeted test for that classification. ad5eac0

ID string `json:"id"`
Name string `json:"name,omitempty"`
Status string `json:"status"`
Value *float64 `json:"value,omitempty"`

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preserved explicitly transported null grader scalar fields and empty payload objects during JSON serialization, with tests asserting value, baselineValue, and deltaFromBaseline remain present as null. ad5eac0

Comment on lines 187 to +190
resultsPath := findGraderFile(logsPath, constants.GraderResultsFilename.String())
if resultsPath == "" {
gradersDataLog.Printf("No grader results found in: %s", logsPath)
return nil
gradersDataLog.Printf("No grader results file found in: %s, trying usage activity summary", logsPath)
return graderResultsFromUsageSummary(logsPath)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the graders presence check to recognize non-empty grader results embedded in usage/activity/summary.json, so --graders no longer filters those runs out before report construction. ad5eac0

Comment thread pkg/cli/audit_report_graders.go Outdated
Comment on lines 208 to 211
@@ -164,19 +210,22 @@ func extractGradersData(logsPath string) *GradersData {
gradersDataLog.Printf("Failed to parse grader results: %v", err)
return nil

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added fallback to the embedded usage activity summary when the first standalone grader results file exists but is unreadable, oversized, or malformed, with Go coverage for the malformed-standalone path. ad5eac0

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs and /tdd to the run-created-at acquisition fix and the grader-result serialization contract.

📋 Key Themes & Highlights

Key Themes

  • The fix follows the diagnosing-bugs discipline well: normalize → retry transient failures only → fall back to Actions metadata → fail loudly with a specific error when acquisition is truly impossible, rather than silently grading an invalid request. Root cause note is honest about not being reproducible locally, and the change is explicitly defensive.
  • Good regression-test coverage: run_created_at.test.cjs covers normalization edge cases (offsets, sub-second truncation, malformed strings), retry-vs-fallback branching, and acquisition failure messaging. audit_report_graders_test.go round-trips a rich payload (zero value, null value, ratio unit, baseline fields, diagnostics) through JSON exactly as the cached JSONL path does — this is precisely the kind of "reproduce-then-regress" test the bug warranted.
  • trace_graders.cjs's "never grade without run metadata" branch is tested end-to-end (operational-value run created-at acquisition describe block), confirming the explicit error result and diagnostics.missingReason contract.

Minor Observations (non-blocking)

  • operational_value_grader.cjs's buildRunSubject now calls parseTimestamp(metadata.createdAt, "run.createdAt") purely for its throwing side effect and discards the parsed epoch, then re-stores the original string as createdAt: metadata.createdAt || null. This works but reads a little surprising — a short comment noting the return value is intentionally unused (validation only) would help future readers per /codebase-design's "make intent explicit" guidance.
  • generate_usage_activity_summary.cjs's parseGraderResults swallows per-candidate read errors into a core.warning and only rethrows if all candidates fail — reasonable, but worth double-checking that a malformed primary file with a valid fallback doesn't mask a real corruption bug from repeat occurrences (currently only surfaced as a warning, no test asserts on this exact multi-candidate-fallback logging path).

Positive Highlights

  • ✅ Clean separation: all created-at logic consolidated into run_created_at.cjs, replacing duplicated ad-hoc retry logic in generate_aw_info.cjs — directly addresses the earlier inline review request to extract this logic.
  • ✅ GraderResult/graderArtifactFullResult field additions are well-documented, explaining why fields must survive serialization unchanged (unit-as-fallback comment is a nice touch).
  • ✅ Constants deduplicated (GRADERS_DIR_NAME etc. shared between JS and Go via comment cross-references).

No blocking issues found — approving.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 81.8 AIC · ⌖ 15 AIC · ⊞ 10.4K
Comment /matt to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the run_created_at.cjs acquisition/normalization logic, the trace_graders.cjs grader-loop gating, operational_value_grader.cjs's buildRunSubject validation, and the Go GraderResult/graderArtifactFullResult serialization contract (including the usage-activity-summary fallback path in audit_report_graders.go / logs_usage_activity.go).

Findings:

  • normalizeRunCreatedAt correctly rejects loose ISO forms before Date.parse, truncates sub-second precision, and returns "" for anything unusable — matches its doc comment and tests.
  • resolveRunCreatedAtForGrading / resolveRunCreatedAtForInfo retry only transient failures (5xx, rate limits, network errors) via withRetry/isTransientError, and fail fast on permanent 4xx — confirmed against error_recovery.cjs.
  • The grader loop in trace_graders.cjs only attempts resolveRunCreatedAtForGrading() when an operational-value grader is actually enabled (operationalValueManifest check), and now emits an explicit error status with diagnostics.missingReason instead of silently grading an invalid request — this is the core fix and looks correct.
  • buildRunSubject now rejects a malformed non-null createdAt via parseTimestamp rather than forwarding it.
  • Go GraderResult/graderArtifactFullResult additions (Source, Implementation, Observation, Diagnostics, BaselineValue, DeltaFromBaseline) round-trip correctly, including null-vs-zero handling in parseGraderValue, and the declared manifest unit is now strictly a fallback rather than clobbering the grader-reported unit.
  • The graderResultsFromUsageSummary fallback (standalone grader_results.json → embedded activity/summary.json copy) is sound and doesn't shadow a good standalone file.
  • Ran the targeted Go tests (go test ./pkg/cli/... -run TestExtractGradersData) — all pass, including the new round-trip and usage-activity-summary fallback tests. JS test execution wasn't possible in this sandbox (no network access for npm install), but the new run_created_at.test.cjs/trace_graders.test.cjs/generate_aw_info.test.cjs cases were read and appear to correctly cover retry/no-retry/fallback/normalization paths.

No blocking issues found in the changed lines. Schema regenerations (schemas/*.json) match the new Go struct fields.

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • codeload.github.com
  • github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "codeload.github.com"
    - "github.com"

See Network Configuration for more information.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 103.1 AIC · ⌖ 13.7 AIC · ⊞ 8.4K

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot
finding and processing the first grader file is ok, no need to process them all
run pr-finisher skill

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

@copilot
finding and processing the first grader file is ok, no need to process them all
run pr-finisher skill

Adjusted the conclusion-job grader summary reader to resolve and process only the first existing grader results file via the shared helper, and updated the helper to stop at the first match. Also ran the PR-finisher flow items available in this environment. ad5eac0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants