Use frontmatter SHA as workflow evaluation identity by Copilot · Pull Request #67300 · github/gh-aw

Copilot · 2026-10-09T23:30:08Z

Prompt-body edits are common, minor changes and should not create a new evaluation identity. The work-queue protocol should distinguish that identity from the immutable Git revision used to route workers.

Evaluation identity: Derive the experiment harness version solely from the compiled frontmatter SHA. A body-only edit now retains the same identity.
Protocol guidance: Clarify that body-only edits do not require repeating the full protocol verification suite. Focused validation still applies, and immutable worker routing remains unchanged.
Regression coverage: Check that changing the body hash leaves the harness identity unchanged.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-10-09T23:39:07Z

@copilot update TLA+ specs, docs

pelikhan · 2026-10-09T23:40:43Z

@copilot merge main and fix conflicts

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-10-09T23:54:01Z

@copilot update TLA+ specs, docs

Added a focused TLA+ model for frontmatter-based evaluation identity and immutable worker revision routing, wired it into the formal suite, and documented the distinction. TLC passes. Commit: d87dda6.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

…ueue-protocol-again # Conflicts: # specs/work-queue/README.md # specs/work-queue/check.sh Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-10-10T00:05:03Z

@copilot merge main and fix conflicts

Merged main and resolved both formal-suite conflicts while retaining the HarnessIdentity and QueueBootstrap models and checks. The merge commit is 5be9763.

Copilot

🟡 Changes recommended

The selected frontmatter hash remains body-sensitive for inlined imports and template expressions, invalidating the promised stable evaluation identity.

2 open findings

What changed in this PR

Changes experiment harness identity to use the compiled frontmatter hash while separating immutable worker routing.

Changes:

Emits frontmatter-only harness versions and regenerates 42 workflow locks.
Adds unit and TLA+ regression coverage.
Documents evaluation identity and routing semantics.

Validation: All regenerated harness values match lock metadata. However, the existing FrontmatterHash can include body content, so body-only identity stability is not guaranteed.

File	Description
`pkg/workflow/compiler_experiments.go`	Uses `FrontmatterHash` as harness identity.
`pkg/workflow/compiler_experiments_test.go`	Updates identity unit coverage.
`specs/work-queue/HarnessIdentity.tla`	Models identity and routing behavior.
`specs/work-queue/HarnessIdentity.cfg`	Configures the new TLA+ model.
`specs/work-queue/check.sh`	Adds the model to TLC checks.
`specs/work-queue/README.md`	Documents focused model execution.
`docs/src/content/docs/specs/work-queue-specification.md`	Documents identity semantics.
`.github/workflows/agent-performance-analyzer.lock.yml`	Regenerates harness identity.
`.github/workflows/agent-persona-explorer.lock.yml`	Regenerates harness identity.
`.github/workflows/audit-workflows.lock.yml`	Regenerates harness identity.
`.github/workflows/aw-failure-investigator.lock.yml`	Regenerates harness identity.
`.github/workflows/blog-auditor.lock.yml`	Regenerates harness identity.
`.github/workflows/breaking-change-checker.lock.yml`	Regenerates harness identity.
`.github/workflows/ci-coach.lock.yml`	Regenerates harness identity.
`.github/workflows/copilot-agent-analysis.lock.yml`	Regenerates harness identity.
`.github/workflows/copilot-pr-nlp-analysis.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-agentrx-trace-optimizer.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-architecture-diagram.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-astrostylelite-markdown-spellcheck.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-cache-strategy-analyzer.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-caveman-optimizer.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-code-metrics.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-community-attribution.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-compiler-quality.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-doc-healer.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-doc-updater.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-fact.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-issues-report.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-news.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-rendering-scripts-verifier.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-safe-output-optimizer.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-security-red-team.lock.yml`	Regenerates harness identity.
`.github/workflows/daily-semgrep-scan.lock.yml`	Regenerates harness identity.
`.github/workflows/dataflow-pr-discussion-dataset.lock.yml`	Regenerates harness identity.
`.github/workflows/deep-report.lock.yml`	Regenerates harness identity.
`.github/workflows/dependabot-go-checker.lock.yml`	Regenerates harness identity.
`.github/workflows/gpclean.lock.yml`	Regenerates harness identity.
`.github/workflows/issue-arborist.lock.yml`	Regenerates harness identity.
`.github/workflows/plan.lock.yml`	Regenerates harness identity.
`.github/workflows/smoke-copilot-aoai-apikey.lock.yml`	Regenerates harness identity.
`.github/workflows/smoke-copilot-aoai-entra.lock.yml`	Regenerates harness identity.
`.github/workflows/smoke-copilot-sub-agents.lock.yml`	Regenerates harness identity.
`.github/workflows/smoke-copilot.lock.yml`	Regenerates harness identity.
`.github/workflows/smoke-gemini.lock.yml`	Regenerates harness identity.
`.github/workflows/smoke-project.lock.yml`	Regenerates harness identity.
`.github/workflows/smoke-temporary-id.lock.yml`	Regenerates harness identity.
`.github/workflows/test-quality-sentinel.lock.yml`	Regenerates harness identity.
`.github/workflows/typist.lock.yml`	Regenerates harness identity.
`.github/workflows/weekly-blog-post-writer.lock.yml`	Regenerates harness identity.

🧠 Review effort: Balanced

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Copilot · 2026-10-10T03:07:18Z

-	default:
-		return data.FrontmatterHash + ":" + data.BodyHash
 	}
+	return data.FrontmatterHash


The default frontmatter identity now hashes FrontmatterYAML directly and never falls back to the body-sensitive compiler freshness hash. The real CompileWorkflow regression verifies inline-body edits change lock freshness hashes while preserving the experiment identity. Commit: 1ed10d7.

Copilot · 2026-10-10T03:07:18Z

+    /\ bodyHash' \in BodyHashes \ {bodyHash}
+    /\ sourceRevision' \in Revisions \ usedRevisions
+    /\ usedRevisions' = usedRevisions \cup {sourceRevision'}
+    /\ UNCHANGED <<frontmatterHash, assignmentRevision>>


The standalone TLA+ identity model was removed per the YAGNI feedback; the Go regression now exercises actual compilation and verifies the body-sensitive freshness hash can change while identity remains stable. Commit: 1ed10d7.

github-actions · 2026-10-10T00:47:20Z

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

No ADR enforcement needed for PR #67300: the PR does not carry the 'implementation' label and adds only 7 new lines in default business logic directories (threshold: 100).

🏗️ ADR gate enforced by Design Decision Gate 🏗️

github-actions · 2026-10-10T00:47:25Z

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

github-actions · 2026-10-10T00:47:30Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

github-actions · 2026-10-10T00:47:39Z

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

github-actions · 2026-10-10T00:47:40Z

✅ Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #67300

github-actions

Request changes

This still does not make evaluation identity "frontmatter SHA only" in the actual compiler. experimentHarnessVersion() now returns FrontmatterHash, but that hash still incorporates body-only changes for inlined-imports: true workflows and for body env./vars. expressions. The PR therefore ships a behavior/doc/spec mismatch instead of the advertised identity split.

Blocking theme

The implementation, tests, docs, and new TLA model now all assume body-only edits preserve evaluation identity, but the current hash pipeline does not. That means users will still see harness identity churn on some prompt-body edits, and the new formal model is proving a different system than the one we execute.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 25.5 AIC · ⌖ 7.99 AIC · ⊞ 19.5K
_{Comment /review to run again}

github-actions

L1-76: yagni: standalone 76-line TLA identity model plus its config, runner wiring, and README invocation for a one-field hash selector. Keep the focused Go regression test that varies BodyHash; it directly specifies the shipped behavior.

net: -102 lines possible.

Generated by ✂️ Ponytail Reviewer for #67300 · codex · gpt56 · 8.45 AIC · ⌖ 6.54 AIC · ⊞ 13.5K
_{Comment /ponytail to run again}

github-actions · 2026-10-10T00:52:00Z

@@ -0,0 +1,76 @@
+-------------------------- MODULE HarnessIdentity --------------------------


L1-76: yagni: standalone 76-line TLA identity model plus its config, runner wiring, and README invocation for a one-field hash selector. Keep the focused Go regression test that varies BodyHash; it directly specifies the shipped behavior.

Removed the standalone model, its config, runner wiring, and README invocation. The focused Go test now covers this behavior through the real compiler path. Commit: 1ed10d7.

github-actions

Skills-Based Review 🧠

Applied /codebase-design — requesting changes on the core correctness claim.

📋 Key Themes & Highlights

Key Themes

Invariant doesn't hold universally: The PR's central claim — "a body-only edit now retains the same identity" — is false for the ~18 workflows in this repo using inlined-imports: true (e.g. daily-fact.md), because FrontmatterHash folds the full body into the hash in that mode. Both existing inline Copilot review comments on compiler_experiments.go:691 and HarnessIdentity.tla:29 correctly flag this; the new TLA+ property BodyEditPreservesIdentity is proven against a model that assumes frontmatterHash is always body-independent, which doesn't match the implementation it's meant to certify.
No changeset: This PR changes compiled output for 49 workflows (GH_AW_HARNESS_VERSION format) but adds no .changeset/*.md entry, unlike the project's typical practice for behavior-affecting compiler changes.

Positive Highlights

✅ Clean, minimal simplification of experimentHarnessVersion — the switch → if reduction is easy to follow.
✅ Good regression test coverage distinguishing frontmatter-hash-only vs. unknown cases (TestExperimentHarnessVersionUsesOnlyFrontmatterHash).
✅ Documentation update in work-queue-specification.md is clear and links correctly to the new formal model section.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 91.6 AIC · ⌖ 15.4 AIC · ⊞ 10.1K
_{Comment /matt to run again}

github-actions · 2026-10-10T01:27:10Z

Thanks for this contribution! The change to use frontmatter SHA as workflow evaluation identity is focused and includes test coverage. Looks ready for review.

Generated by ✅ Contribution Check · copilot · auto · 48.5 AIC · ⌖ 0.613 AIC · ⊞ 9.2K · ◷

Done

pelikhan · 2026-10-10T01:50:50Z

@copilot the user should be allowed to define the level of identity fingerprinting needed. full hash, front matter hash, workflow id.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-10-10T02:11:46Z

@copilot the user should be allowed to define the level of identity fingerprinting needed. full hash, front matter hash, workflow id.

Added experiments.identity modes full, frontmatter (default), and workflow, with the compiler, tests, TLA+ model, and docs updated. Commit: eb37404.

gh-aw-bot · 2026-10-10T02:45:13Z

@copilot address the following outstanding work in one pass:

Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
Review (pkg/workflow/compiler_experiments.go:691): FrontmatterHash is not a pure frontmatter-only value: its computation includes the entire body for inlined-imports: true and body env/vars expressions otherwise (pkg/parser/frontmatter_hash.go:123-137,885-890). For example, .github/workflows/daily-fact.md has both experiments and inlined-imports: true, so any prompt-only edit still changes this value and therefore the harness identity. This contradicts the promised body-edit stability; derive evaluation identity from a separate pure-frontmatter hash while retaining the existing hybrid hash for lock freshness, and cover the real c... - Use frontmatter SHA as workflow evaluation identity #67300 (comment)
Review (specs/work-queue/HarnessIdentity.tla:29): This transition assumes every body edit leaves frontmatterHash unchanged, so BodyEditPreservesIdentity is true by construction. The compiler's current frontmatter hash changes for every body edit under inlined-imports: true and for changed body env/vars expressions, so the model excludes the implementation counterexample it is meant to check. Represent the pure evaluation hash separately from the compiler freshness hash (or model the body-sensitive hash transition) so TLC verifies the real boundary. - Use frontmatter SHA as workflow evaluation identity #67300 (comment)
Review (specs/work-queue/HarnessIdentity.tla:1): L1-76: yagni: standalone 76-line TLA identity model plus its config, runner wiring, and README invocation for a one-field hash selector. Keep the focused Go regression test that varies BodyHash; it directly specifies the shipped behavior. - Use frontmatter SHA as workflow evaluation identity #67300 (comment)
Fix failing check build (FAILURE): https://github.com/github/gh-aw/actions/runs/38016031690/job/114108028529.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: eb37404
Sous-chef work: 08b6f2f195e2dfe47e1d14807b6e17cb58359bae042355d75138b4f5c7c64efe 166a205f04f0e8787ab106be8428b7107cdb4e1e448e76d4cc4b305fd77e86b4 d5d8e013edf7e83d4ab89f81160577849abd74f8dc3158bac876a0d44ce960b4 f7883e8c8105930e955354caa741d783bcca067e66097fe9612fb72b90823998
Sous-chef state: c477b34288b74baca5e9f0f86e7eb3e443dadaa728973f4be1f2a56a23e0f996

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 3.85 AIC · ⌖ 8.87 AIC · ⊞ 1K · ◷
_{Comment /souschef to run again}

…ueue-protocol-again Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot · 2026-10-10T03:07:18Z

@copilot address the following outstanding work in one pass:

Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Committed and pushed as 1ed10d7. make merge-main succeeded; the pure-frontmatter identity regression and model/doc cleanup are in place, including removal of the invalid docs link that caused the reported build failure. Local docs build could not install dependencies because the configured npm registry was unreachable, and the final progress gate reported pre-existing custom Go linter findings in latest main.

Use frontmatter identity for evaluation and clarify queue verification

6975b79

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI assigned Copilot and pelikhan Oct 9, 2026

Copilot AI changed the title ~~Use frontmatter SHA for workflow evaluation identity~~ Use frontmatter SHA as workflow evaluation identity Oct 9, 2026

Copilot created this pull request from a session on behalf of pelikhan October 9, 2026 23:30 View session

Copilot finished work on behalf of pelikhan October 9, 2026 23:30

Copilot AI requested a review from pelikhan October 9, 2026 23:30

Copilot started work on behalf of pelikhan October 9, 2026 23:39 View session

Specify workflow evaluation identity in TLA+

d87dda6

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Apply remaining changes

74fc782

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan October 9, 2026 23:54

Copilot started work on behalf of pelikhan October 9, 2026 23:54 View session

Merge remote-tracking branch 'origin/main' into copilot/review-work-q…

5be9763

…ueue-protocol-again # Conflicts: # specs/work-queue/README.md # specs/work-queue/check.sh Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan October 10, 2026 00:05

pelikhan marked this pull request as ready for review October 10, 2026 00:15

Copilot AI balanced review requested due to automatic review settings October 10, 2026 00:15

Copilot started reviewing on behalf of pelikhan October 10, 2026 00:16 View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

github-actions Bot requested changes Oct 10, 2026

View reviewed changes

github-actions Bot reviewed Oct 10, 2026

View reviewed changes

github-actions Bot mentioned this pull request Oct 10, 2026

[aw] No-Op Runs #65943

Open

github-actions Bot previously requested changes Oct 10, 2026

View reviewed changes

This was referenced Oct 10, 2026

[PR Triage Report] PR triage report: no eligible Copilot PRs under cooldown #67311

Closed

[Contribution Check Report] Contribution Check — 2026-10-10 #67317

Closed

Copilot started work on behalf of pelikhan October 10, 2026 01:51 View session

Configure experiment identity fingerprints

eb37404

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan October 10, 2026 02:12

github-actions Bot mentioned this pull request Oct 10, 2026

[pr-sous-chef] PR Sous Chef Report – 2026-10-10T02:09:05Z #67318

Closed

Copilot started work on behalf of gh-aw-bot October 10, 2026 02:45 View session

Copilot AI and others added 2 commits October 10, 2026 02:52

Merge remote-tracking branch 'origin/main' into copilot/review-work-q…

f863d9c

…ueue-protocol-again Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Fix pure frontmatter experiment identity

1ed10d7

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot finished work on behalf of gh-aw-bot October 10, 2026 03:08

Copilot AI requested a review from gh-aw-bot October 10, 2026 03:08

pelikhan closed this Oct 10, 2026

		@@ -0,0 +1,76 @@
		-------------------------- MODULE HarnessIdentity --------------------------

Conversation

Copilot AI commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelikhan commented Oct 9, 2026

Uh oh!

pelikhan commented Oct 9, 2026

Uh oh!

Copilot AI commented Oct 9, 2026

Uh oh!

Copilot AI commented Oct 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Copilot AI Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Request changes

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Key Themes

Positive Highlights

Uh oh!

github-actions Bot commented Oct 10, 2026

Uh oh!

pelikhan commented Oct 10, 2026

Uh oh!

Copilot AI commented Oct 10, 2026

Uh oh!

gh-aw-bot commented Oct 10, 2026

Uh oh!

Copilot AI commented Oct 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Oct 9, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading

github-actions Bot commented Oct 10, 2026 •

edited

Loading