Repository navigation
Conversation
|
@copilot Make sure graders information is collected fully in the conclusion job in the usage artifact .json payload |
…esult contract in logs Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
… schemas Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
| */ | ||
| async function resolveRunCreatedAt(core, ctx, githubClient) { | ||
| let lastError = "workflow run creation time is unavailable"; | ||
| for (let attempt = 1; attempt <= RUN_CREATED_AT_ATTEMPTS; attempt++) { |
There was a problem hiding this comment.
Switched to withRetry from error_recovery.cjs (shared RUN_CREATED_AT_RETRY_CONFIG, transient-only shouldRetry), replacing the hand-rolled retry loop. c25ee41
| const { ERR_CONFIG, ERR_SYSTEM } = require("./error_codes.cjs"); | ||
|
|
||
| // Retries guard the operational-value grader against transient run-metadata lookup failures. | ||
| const RUN_CREATED_AT_ATTEMPTS = 3; |
There was a problem hiding this comment.
@copilot move run create at logic into own file, add tests
Move constants to constants
There was a problem hiding this comment.
All created-at logic now lives in run_created_at.cjs (normalizeRunCreatedAt, fetchRunCreatedAt, resolveRunCreatedAtForInfo, resolveRunCreatedAtForGrading); generate_aw_info.cjs and trace_graders.cjs just call it. Retry constants moved to constants.cjs, and the tests moved into run_created_at.test.cjs. c25ee41
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
gh aw logs
…e usage summary Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
The conclusion job's |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
@copilot there are probably helper functions to resolve files in multiple potential locations, use those helpers |
|
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff
|
|
✅ Ponytail Reviewer completed successfully! Lean already. Ship. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "ab.chatgpt.com"See Network Configuration for more information.
|
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅ Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "registry.npmjs.org"See Network Configuration for more information.
|
|
Warning Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding. What happenedThe threat detection engine failed to produce results. Review the workflow run logs for details.
|
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was no shared JS helper for this (only duplicated |
🏗️ ADR Required — draft added for PR #60635An ADR was required for this PR because it adds more than 100 lines in business-logic directories ( Evidence reviewed
Gate resultNo existing ADR for this decision was found in the PR body or on the branch. I generated and committed a draft ADR:
Inferred decisionThis PR makes one architectural decision: operational-value grading must never proceed without a valid workflow run creation timestamp, and grader results must preserve the full result contract across artifact and CLI boundaries. Next actionPlease review and refine the draft ADR, especially the rationale and trade-offs, before merging this PR.
|
There was a problem hiding this comment.
🟡 Changes recommended
Null fields and empty diagnostics remain lossy, while timestamp validation and fallback paths have correctness gaps.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Fixes #60633 by improving run-created-at acquisition and preserving operational-value grader data in logs.
Changes:
- Adds timestamp normalization, retry, fallback, and explicit failure handling.
- Preserves rich grader fields and embeds results in usage summaries.
- Adds tests, schemas, documentation, and a patch changeset.
File summaries
| File | Description |
|---|---|
schemas/logs.schema.json |
Adds rich grader fields. |
schemas/logs-jsonl.schema.json |
Extends JSONL grader schemas. |
schemas/audit.schema.json |
Extends audit grader schema. |
pkg/cli/logs_usage_activity.go |
Parses embedded grader results. |
pkg/cli/audit_report_graders.go |
Expands grader serialization and fallback loading. |
pkg/cli/audit_report_graders_test.go |
Tests rich result round-tripping and fallback. |
docs/src/content/docs/specs/graders-specification.md |
Defines acquisition and serialization requirements. |
docs/src/content/docs/reference/artifacts.md |
Documents embedded grader results. |
actions/setup/js/trace_graders.test.cjs |
Tests acquisition failure handling. |
actions/setup/js/trace_graders.cjs |
Resolves metadata before operational grading. |
actions/setup/js/run_created_at.test.cjs |
Tests timestamp acquisition and normalization. |
actions/setup/js/run_created_at.cjs |
Adds shared timestamp lookup logic. |
actions/setup/js/operational_value_grader.test.cjs |
Tests created-at validation. |
actions/setup/js/operational_value_grader.cjs |
Validates the run creation timestamp. |
actions/setup/js/generate_usage_activity_summary.test.cjs |
Tests grader-result embedding. |
actions/setup/js/generate_usage_activity_summary.cjs |
Embeds grader results in usage summaries. |
actions/setup/js/generate_aw_info.test.cjs |
Tests normalization and retries. |
actions/setup/js/generate_aw_info.cjs |
Uses shared created-at resolution. |
actions/setup/js/constants.cjs |
Adds retry and grader-path constants. |
.changeset/operational-value-run-created-at.md |
Records the patch release. |
Review details
Suppressed comments (2)
pkg/cli/audit_report_graders.go:46
- Explicit
nullbaseline fields are converted to nil pointers and then omitted here. The source operational-value result always carries these fields, and the new specification requires them to survive republishing, so an unavailable baseline must serialize as"baselineValue": nulland"deltaFromBaseline": nullrather than disappearing.
BaselineValue *float64 `json:"baselineValue,omitempty"`
DeltaFromBaseline *float64 `json:"deltaFromBaseline,omitempty"`
pkg/cli/audit_report_graders.go:44
- An operational-value evaluator may legitimately emit
diagnostics: {}(the updated artifact documentation shows this shape), but an empty map satisfiesomitemptyand disappears fromgh aw logs. This still violates the new requirement to preserve diagnostics and again makes an empty diagnostic payload indistinguishable from transport loss. Represent presence separately or otherwise ensure an explicit empty object survives marshaling.
Diagnostics map[string]any `json:"diagnostics,omitempty"`
- Files reviewed: 23/23 changed files
- Comments generated: 5
- Review effort level: Balanced
| if (!ISO_8601_DATE_TIME.test(trimmed)) return ""; | ||
| const parsed = Date.parse(trimmed); | ||
| if (!Number.isFinite(parsed)) return ""; | ||
| return new Date(Math.trunc(parsed / 1000) * 1000).toISOString().replace(/\.\d{3}Z$/, "Z"); |
There was a problem hiding this comment.
Fixed timestamp normalization to validate captured calendar/time/offset components before Date.parse, so malformed dates like February 30 are rejected instead of normalized. Added coverage for invalid dates and leap days. ad5eac0
| shouldRetry: error => { | ||
| const status = Number(error?.status ?? error?.response?.status); | ||
| return (Number.isInteger(status) && status >= 500) || isTransientError(error); | ||
| }, |
There was a problem hiding this comment.
Updated the shared withRetry config to treat status-only HTTP 408 errors as retryable and added a targeted test for that classification. ad5eac0
| ID string `json:"id"` | ||
| Name string `json:"name,omitempty"` | ||
| Status string `json:"status"` | ||
| Value *float64 `json:"value,omitempty"` |
There was a problem hiding this comment.
Preserved explicitly transported null grader scalar fields and empty payload objects during JSON serialization, with tests asserting value, baselineValue, and deltaFromBaseline remain present as null. ad5eac0
| resultsPath := findGraderFile(logsPath, constants.GraderResultsFilename.String()) | ||
| if resultsPath == "" { | ||
| gradersDataLog.Printf("No grader results found in: %s", logsPath) | ||
| return nil | ||
| gradersDataLog.Printf("No grader results file found in: %s, trying usage activity summary", logsPath) | ||
| return graderResultsFromUsageSummary(logsPath) |
There was a problem hiding this comment.
Updated the graders presence check to recognize non-empty grader results embedded in usage/activity/summary.json, so --graders no longer filters those runs out before report construction. ad5eac0
| @@ -164,19 +210,22 @@ func extractGradersData(logsPath string) *GradersData { | |||
| gradersDataLog.Printf("Failed to parse grader results: %v", err) | |||
| return nil | |||
There was a problem hiding this comment.
Added fallback to the embedded usage activity summary when the first standalone grader results file exists but is unreadable, oversized, or malformed, with Go coverage for the malformed-standalone path. ad5eac0
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /diagnosing-bugs and /tdd to the run-created-at acquisition fix and the grader-result serialization contract.
📋 Key Themes & Highlights
Key Themes
- The fix follows the diagnosing-bugs discipline well: normalize → retry transient failures only → fall back to Actions metadata → fail loudly with a specific error when acquisition is truly impossible, rather than silently grading an invalid request. Root cause note is honest about not being reproducible locally, and the change is explicitly defensive.
- Good regression-test coverage:
run_created_at.test.cjscovers normalization edge cases (offsets, sub-second truncation, malformed strings), retry-vs-fallback branching, and acquisition failure messaging.audit_report_graders_test.goround-trips a rich payload (zero value, null value, ratio unit, baseline fields, diagnostics) through JSON exactly as the cached JSONL path does — this is precisely the kind of "reproduce-then-regress" test the bug warranted. trace_graders.cjs's "never grade without run metadata" branch is tested end-to-end (operational-value run created-at acquisitiondescribe block), confirming the explicit error result anddiagnostics.missingReasoncontract.
Minor Observations (non-blocking)
operational_value_grader.cjs'sbuildRunSubjectnow callsparseTimestamp(metadata.createdAt, "run.createdAt")purely for its throwing side effect and discards the parsed epoch, then re-stores the original string ascreatedAt: metadata.createdAt || null. This works but reads a little surprising — a short comment noting the return value is intentionally unused (validation only) would help future readers per/codebase-design's "make intent explicit" guidance.generate_usage_activity_summary.cjs'sparseGraderResultsswallows per-candidate read errors into acore.warningand only rethrows if all candidates fail — reasonable, but worth double-checking that a malformed primary file with a valid fallback doesn't mask a real corruption bug from repeat occurrences (currently only surfaced as a warning, no test asserts on this exact multi-candidate-fallback logging path).
Positive Highlights
- ✅ Clean separation: all created-at logic consolidated into
run_created_at.cjs, replacing duplicated ad-hoc retry logic ingenerate_aw_info.cjs— directly addresses the earlier inline review request to extract this logic. - ✅
GraderResult/graderArtifactFullResultfield additions are well-documented, explaining why fields must survive serialization unchanged (unit-as-fallback comment is a nice touch). - ✅ Constants deduplicated (
GRADERS_DIR_NAMEetc. shared between JS and Go via comment cross-references).
No blocking issues found — approving.
Warning
Firewall blocked 1 domain
The following domain was blocked by the firewall during workflow execution:
registry.npmjs.org
To allow these domains, add them to the network.allowed list in your workflow frontmatter:
network:
allowed:
- defaults
- "registry.npmjs.org"See Network Configuration for more information.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 81.8 AIC · ⌖ 15 AIC · ⊞ 10.4K
Comment /matt to run again
There was a problem hiding this comment.
Reviewed the run_created_at.cjs acquisition/normalization logic, the trace_graders.cjs grader-loop gating, operational_value_grader.cjs's buildRunSubject validation, and the Go GraderResult/graderArtifactFullResult serialization contract (including the usage-activity-summary fallback path in audit_report_graders.go / logs_usage_activity.go).
Findings:
normalizeRunCreatedAtcorrectly rejects loose ISO forms beforeDate.parse, truncates sub-second precision, and returns""for anything unusable — matches its doc comment and tests.resolveRunCreatedAtForGrading/resolveRunCreatedAtForInforetry only transient failures (5xx, rate limits, network errors) viawithRetry/isTransientError, and fail fast on permanent 4xx — confirmed againsterror_recovery.cjs.- The grader loop in
trace_graders.cjsonly attemptsresolveRunCreatedAtForGrading()when an operational-value grader is actually enabled (operationalValueManifestcheck), and now emits an expliciterrorstatus withdiagnostics.missingReasoninstead of silently grading an invalid request — this is the core fix and looks correct. buildRunSubjectnow rejects a malformed non-nullcreatedAtviaparseTimestamprather than forwarding it.- Go
GraderResult/graderArtifactFullResultadditions (Source,Implementation,Observation,Diagnostics,BaselineValue,DeltaFromBaseline) round-trip correctly, including null-vs-zero handling inparseGraderValue, and the declared manifestunitis now strictly a fallback rather than clobbering the grader-reported unit. - The
graderResultsFromUsageSummaryfallback (standalonegrader_results.json→ embeddedactivity/summary.jsoncopy) is sound and doesn't shadow a good standalone file. - Ran the targeted Go tests (
go test ./pkg/cli/... -run TestExtractGradersData) — all pass, including the new round-trip and usage-activity-summary fallback tests. JS test execution wasn't possible in this sandbox (no network access fornpm install), but the newrun_created_at.test.cjs/trace_graders.test.cjs/generate_aw_info.test.cjscases were read and appear to correctly cover retry/no-retry/fallback/normalization paths.
No blocking issues found in the changed lines. Schema regenerations (schemas/*.json) match the new Go struct fields.
Warning
Firewall blocked 2 domains
The following domains were blocked by the firewall during workflow execution:
codeload.github.comgithub.com
To allow these domains, add them to the network.allowed list in your workflow frontmatter:
network:
allowed:
- defaults
- "codeload.github.com"
- "github.com"See Network Configuration for more information.
🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 103.1 AIC · ⌖ 13.7 AIC · ⊞ 8.4K
|
@copilot |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Adjusted the conclusion-job grader summary reader to resolve and process only the first existing grader results file via the shared helper, and updated the helper to stop at the first match. Also ran the PR-finisher flow items available in this environment. ad5eac0 |
Operational-value grading was failing at two boundaries: graders received
run.createdAt: nulland silently graded an invalid request, andgh aw logs --artifacts graderscollapsed the rich grader result into a lossy summary (droppingsource,implementation,observation,diagnostics, and baseline fields, and rewritingunitfromratiotocount). Downstream consumers could not distinguish missing evidence from transport loss.Created-at acquisition (JS runtime)
run_created_at.cjs(new) — sharednormalizeRunCreatedAt(): strict ISO-8601 validation beforeDate.parse, normalization to second-granularity UTC (the formatparseTimestamprequires),""for anything unusable.generate_aw_info.cjs—resolveRunCreatedAt()fails fast when no authenticated client is available, retries transientgetWorkflowRunfailures with incremental backoff, breaks early on permanent 4xx (except 408/429), and normalizes beforecore.setOutput("run_created_at", …).trace_graders.cjs—resolveRunCreatedAtForGrading()prefersGH_AW_RUN_CREATED_AT, falls back to Actions run metadata, and names the missing input when neither is usable. The grader loop now enforces "never grade without run metadata": an operational-value grader with no resolved created-at yields an explicit error rather than an invalid request.{ "id": "operational-value", "status": "error", "unit": "ratio", "error": "grader operational-value runtime error: workflow run creation time is unavailable and cannot be read from Actions run metadata (missing an authenticated GitHub client)", "diagnostics": { "missingReason": "run created-at acquisition failed" } }operational_value_grader.cjs—buildRunSubjectrejects a malformed non-nullcreatedAtinstead of passing it through.Serialization contract (Go)
pkg/cli/audit_report_graders.go—GraderResultcarriessource,implementation(incl.digest),observation,diagnostics,baselineValue, anddeltaFromBaseline, read fromgrader_results.json. The manifestunitis now strictly a fallback, so aratiometric no longer degrades tocountduring serialization. Free-form payloads usemap[string]any(notjson.RawMessage) so the generated schemas render as objects rather than byte arrays.schemas/audit.schema.json,schemas/logs.schema.json,schemas/logs-jsonl.schema.json.Tests, spec, changeset
ratiounit, maturity fields, baseline fields, diagnostics.graders-specification.md§7 (acquisition requirement) and §8.4 (serialization contract) made normative; patch changeset added.Note on the root cause
Compiler wiring (
GH_AW_INFO_FETCH_RUN_CREATED_AT, the activationrun_created_atoutput, andGH_AW_RUN_CREATED_ATon the graders step) is already correct and the productionnullcould not be reproduced locally, so these changes are defensive: normalize, retry, fall back, and — when all of that fails — report the acquisition failure explicitly instead of emitting anunavailableresult with an invalid request. Note thatactions: readis scoped to the activation job by design, so the graders-step API fallback may 403; that path is exactly what the new error result surfaces.