Skip to content

Commit eb37404

Browse files
Copilotpelikhan
andauthored
Configure experiment identity fingerprints
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
1 parent 5be9763 commit eb37404

10 files changed

Lines changed: 259 additions & 44 deletions

File tree

‎docs/src/content/docs/experimental/experiments-specification.md‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -191,10 +191,29 @@ Notes, rationale blocks, and appendices are **informative** and carry no conform
191191
**MUST** be rejected at compile time with a descriptive error.
192192

193193
**R-SCHEMA-003**: Every key in the `experiments` map, except the reserved `storage` key
194-
(§7.1), **MUST** be an experiment name that matches the regular expression
194+
(§7.1) and `identity` key (§4.1.1), **MUST** be an experiment name that matches the regular expression
195195
`^[a-zA-Z_][a-zA-Z0-9_]*$`. Keys that do not match **MUST** be silently skipped with a
196196
compile-time warning emitted to stderr.
197197

198+
#### 4.1.1 Experiment History Identity
199+
200+
The reserved `identity` key **MUST** accept one of `full`, `frontmatter`, or
201+
`workflow`. If omitted or unrecognized, implementations **MUST** use
202+
`frontmatter`.
203+
204+
| Value | Fingerprint |
205+
|---|---|
206+
| `full` | Compiler freshness hash and complete workflow body hash |
207+
| `frontmatter` | SHA-256 of normalized workflow frontmatter text, excluding the markdown body |
208+
| `workflow` | Workflow ID derived from the workflow filename |
209+
210+
The default `frontmatter` mode **MUST** preserve history across body-only edits,
211+
including workflows that inline imports or use body template expressions.
212+
`full` **MUST** create a new identity when the workflow body changes.
213+
`workflow` **MUST** preserve identity across frontmatter and body edits to the
214+
same workflow ID. Identity selection **MUST NOT** change the immutable source
215+
revision used to route work to a worker.
216+
198217
> **Note (informative)**: The identifier pattern ensures experiment names can be used as
199218
> GitHub Actions step output names and embedded in `${{ experiments.<name> }}` expressions
200219
> without bracket notation.

‎docs/src/content/docs/experimental/experiments.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ on:
3737
engine: copilot
3838

3939
experiments:
40+
identity: frontmatter # full, frontmatter (default), or workflow
4041
style: [concise, detailed]
4142
---
4243

@@ -101,6 +102,23 @@ by variant.
101102
> underscore, followed by letters, digits, or underscores. For example, use
102103
> `style` or `feature_1`. Names that do not match this pattern are ignored.
103104
105+
### Choosing experiment history identity
106+
107+
The reserved `identity` key controls when experiment history is considered to
108+
belong to a new workflow evaluation:
109+
110+
| Value | Identity fingerprint |
111+
|---|---|
112+
| `full` | Compiler freshness hash plus the complete workflow body hash. Prompt or imported-body changes start a new history. |
113+
| `frontmatter` (default) | SHA-256 of the normalized workflow frontmatter text only. Body-only edits keep the same history. |
114+
| `workflow` | Workflow ID derived from the workflow filename. Frontmatter and body edits keep the same history. |
115+
116+
For example, set `experiments.identity: full` when every prompt revision should
117+
be evaluated as a separate harness. Use `workflow` only when history should span
118+
all revisions of that workflow. This choice affects experiment-history grouping;
119+
it does not change the immutable source revision used to route a dispatched
120+
worker.
121+
104122
## Using variants in the prompt
105123

106124
Reference a variant with `${{ experiments.<name> }}`. At runtime, gh-aw

‎docs/src/content/docs/specs/work-queue-specification.md‎

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1205,15 +1205,19 @@ bounded operation are specified in section 7.17.
12051205
| Implementation conformance | Runtime/operator codecs, selector, publisher, binder, and recovery enforce the same contract on real APIs | Native implementations and source-hashed exact conformance results exist; compiler/effects integration and supported-host evidence remain incomplete as tracked in section 9.1 |
12061206
| Operational evidence | Reproducible source/tool identities and explicit passed/violation/incomplete verdicts | Exhausted finite cases are documented separately from two unfinished searches; artifact collection is not a proof |
12071207

1208-
For workflow evaluation identity, use the compiled frontmatter SHA alone.
1209-
Prompt-body changes are routine minor revisions: a changed body hash does not
1210-
create a new evaluation identity or require repeating the entire protocol
1211-
verification suite. Retain the body hash as provenance when available. This
1212-
does not replace the immutable Git revision required to dispatch and
1213-
authenticate an approved worker route, nor does it waive focused validation
1214-
of changed behavior or the protocol-safety acceptance cases below. The
1208+
Workflow authors may choose the fingerprint scope used for experiment
1209+
evaluation identity: `full` (compiler freshness and complete body hashes),
1210+
`frontmatter` (the default, a pure frontmatter hash), or `workflow` (workflow
1211+
ID). In the default mode, prompt-body changes are routine minor revisions: a
1212+
changed body hash does not create a new evaluation identity or require
1213+
repeating the entire protocol verification suite. Retain the body hash as
1214+
provenance when available. This choice does not replace the immutable Git
1215+
revision required to dispatch and authenticate an approved worker route, nor
1216+
does it waive focused validation of changed behavior or the protocol-safety
1217+
acceptance cases below. The
12151218
[formal model](../../../../../specs/work-queue/README.md#evaluation-identity-and-worker-routing)
1216-
checks this separation and the immutability of a dispatched worker's revision.
1219+
checks all three identity scopes and the immutability of a dispatched worker's
1220+
revision.
12171221

12181222
The scheduler admits the largest **fair prefix** allowed by the request and
12191223
capacity limits. It does not promise to fill every physically idle slot.

‎pkg/workflow/compiler_experiments.go‎

Lines changed: 64 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
package workflow
22

33
import (
4+
"crypto/sha256"
5+
"encoding/hex"
46
"encoding/json"
57
"fmt"
68
"math"
@@ -34,12 +36,26 @@ const (
3436
ExperimentsStorageRepo ExperimentStorageMode = "repo"
3537
)
3638

39+
// ExperimentIdentityMode controls which workflow fingerprint scopes experiment history.
40+
type ExperimentIdentityMode string
41+
42+
const (
43+
// ExperimentsIdentityFull includes compiler freshness and body hashes.
44+
ExperimentsIdentityFull ExperimentIdentityMode = "full"
45+
// ExperimentsIdentityFrontmatter hashes only the workflow's frontmatter text.
46+
ExperimentsIdentityFrontmatter ExperimentIdentityMode = "frontmatter"
47+
// ExperimentsIdentityWorkflow uses the workflow identifier without content hashes.
48+
ExperimentsIdentityWorkflow ExperimentIdentityMode = "workflow"
49+
)
50+
3751
// experimentsBranchPrefix is the git branch prefix used when storage: repo is selected.
3852
// Branches are named "experiments/{sanitizedWorkflowID}".
3953
const experimentsBranchPrefix = "experiments"
4054

41-
// experimentsStorageReservedKey is the reserved key in the experiments map that controls storage mode.
42-
const experimentsStorageReservedKey = "storage"
55+
const (
56+
experimentsStorageReservedKey = "storage"
57+
experimentsIdentityReservedKey = "identity"
58+
)
4359

4460
// experimentNamePattern validates experiment names as identifier-style keys.
4561
// Experiment names must match [a-zA-Z_][a-zA-Z0-9_]* so they can be used
@@ -74,8 +90,8 @@ func extractExperimentConfigsFromFrontmatter(frontmatter map[string]any) map[str
7490
}
7591
result := make(map[string]*ExperimentConfig, len(rawMap))
7692
for name, val := range rawMap {
77-
// "storage" is a reserved key that controls persistence mode, not an experiment name.
78-
if name == experimentsStorageReservedKey {
93+
// These reserved keys configure experiment behavior, not experiment names.
94+
if name == experimentsStorageReservedKey || name == experimentsIdentityReservedKey {
7995
continue
8096
}
8197
if !experimentNamePattern.MatchString(name) {
@@ -119,6 +135,27 @@ func extractExperimentsStorageFromFrontmatter(frontmatter map[string]any) Experi
119135
return ExperimentsStorageRepo
120136
}
121137

138+
// extractExperimentsIdentityFromFrontmatter reads the identity mode from the experiments
139+
// map. Frontmatter identity is the default to keep routine prompt edits from resetting history.
140+
func extractExperimentsIdentityFromFrontmatter(frontmatter map[string]any) ExperimentIdentityMode {
141+
raw, ok := frontmatter["experiments"].(map[string]any)
142+
if !ok {
143+
return ExperimentsIdentityFrontmatter
144+
}
145+
if identityRaw, ok := raw[experimentsIdentityReservedKey]; ok {
146+
if identity, ok := identityRaw.(string); ok {
147+
mode := ExperimentIdentityMode(identity)
148+
switch mode {
149+
case ExperimentsIdentityFull, ExperimentsIdentityFrontmatter, ExperimentsIdentityWorkflow:
150+
return mode
151+
default:
152+
experimentsLog.Printf("Unknown experiments identity %q; falling back to %q", identity, ExperimentsIdentityFrontmatter)
153+
}
154+
}
155+
}
156+
return ExperimentsIdentityFrontmatter
157+
}
158+
122159
// experimentsBranchName returns the git branch name used for repo-based experiment storage.
123160
// Format: "experiments/{sanitizedWorkflowID}"
124161
func experimentsBranchName(workflowID string) string {
@@ -685,10 +722,31 @@ func (c *Compiler) generatePickExperimentStep(data *WorkflowData, experimentName
685722
}
686723

687724
func experimentHarnessVersion(data *WorkflowData) string {
688-
if data.FrontmatterHash == "" {
725+
if data == nil {
726+
return "unknown"
727+
}
728+
switch data.ExperimentsIdentity {
729+
case ExperimentsIdentityFull:
730+
if data.FrontmatterHash == "" || data.BodyHash == "" {
731+
return "unknown"
732+
}
733+
return data.FrontmatterHash + ":" + data.BodyHash
734+
case ExperimentsIdentityWorkflow:
735+
if data.WorkflowID == "" {
736+
return "unknown"
737+
}
738+
return data.WorkflowID
739+
default:
740+
frontmatter := strings.TrimSpace(strings.ReplaceAll(data.FrontmatterYAML, "\r\n", "\n"))
741+
if frontmatter != "" {
742+
hash := sha256.Sum256([]byte(frontmatter))
743+
return hex.EncodeToString(hash[:])
744+
}
745+
if data.FrontmatterHash != "" {
746+
return data.FrontmatterHash
747+
}
689748
return "unknown"
690749
}
691-
return data.FrontmatterHash
692750
}
693751

694752
// generateExperimentArtifactUploadStep generates the artifact upload step shared by both storage modes.

‎pkg/workflow/compiler_experiments_test.go‎

Lines changed: 89 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
package workflow
44

55
import (
6+
"bytes"
67
"encoding/json"
78
"os"
89
"path/filepath"
@@ -171,12 +172,75 @@ func TestGenerateExperimentSteps_SpecJSON(t *testing.T) {
171172

172173
func TestExperimentHarnessVersionUsesOnlyFrontmatterHash(t *testing.T) {
173174
assert.Equal(t, "frontmatter-hash", experimentHarnessVersion(&WorkflowData{FrontmatterHash: "frontmatter-hash", BodyHash: "body-hash"}))
174-
assert.Equal(t, "frontmatter-hash", experimentHarnessVersion(&WorkflowData{FrontmatterHash: "frontmatter-hash", BodyHash: "different-body"}))
175+
frontmatter := "experiments:\n style: [short, long]"
176+
assert.Equal(t,
177+
experimentHarnessVersion(&WorkflowData{FrontmatterYAML: frontmatter, FrontmatterHash: "frontmatter-hash", BodyHash: "body-hash"}),
178+
experimentHarnessVersion(&WorkflowData{FrontmatterYAML: frontmatter, FrontmatterHash: "changed-by-inline-body", BodyHash: "different-body"}),
179+
"pure frontmatter identity must ignore compiler freshness and body hashes",
180+
)
175181
assert.Equal(t, "frontmatter-hash", experimentHarnessVersion(&WorkflowData{FrontmatterHash: "frontmatter-hash"}))
176182
assert.Equal(t, "unknown", experimentHarnessVersion(&WorkflowData{BodyHash: "body-hash"}))
177183
assert.Equal(t, "unknown", experimentHarnessVersion(&WorkflowData{}))
178184
}
179185

186+
func TestExperimentHarnessVersionIdentityModes(t *testing.T) {
187+
base := &WorkflowData{
188+
WorkflowID: "daily-fact",
189+
FrontmatterYAML: "experiments:\n style: [short, long]",
190+
FrontmatterHash: "compiler-freshness-hash",
191+
BodyHash: "body-hash",
192+
ExperimentsIdentity: ExperimentsIdentityFrontmatter,
193+
}
194+
frontmatterVersion := experimentHarnessVersion(base)
195+
bodyEdit := *base
196+
bodyEdit.FrontmatterHash = "freshness-hash-changed-by-body"
197+
bodyEdit.BodyHash = "different-body-hash"
198+
assert.Equal(t, frontmatterVersion, experimentHarnessVersion(&bodyEdit))
199+
200+
full := *base
201+
full.ExperimentsIdentity = ExperimentsIdentityFull
202+
fullVersion := experimentHarnessVersion(&full)
203+
full.BodyHash = "different-body-hash"
204+
assert.NotEqual(t, fullVersion, experimentHarnessVersion(&full))
205+
206+
workflow := *base
207+
workflow.ExperimentsIdentity = ExperimentsIdentityWorkflow
208+
assert.Equal(t, "daily-fact", experimentHarnessVersion(&workflow))
209+
workflow.BodyHash = "different-body-hash"
210+
workflow.FrontmatterHash = "different-frontmatter-hash"
211+
assert.Equal(t, "daily-fact", experimentHarnessVersion(&workflow))
212+
}
213+
214+
func TestCompileWorkflowFrontmatterIdentityIgnoresInlinedBodyEdits(t *testing.T) {
215+
compile := func(body string) string {
216+
t.Helper()
217+
dir := t.TempDir()
218+
workflowPath := filepath.Join(dir, "harness.md")
219+
content := `---
220+
on: workflow_dispatch
221+
strict: false
222+
engine: copilot
223+
inlined-imports: true
224+
experiments:
225+
style: [short, long]
226+
---
227+
228+
` + body + "\n"
229+
require.NoError(t, os.WriteFile(workflowPath, []byte(content), 0600))
230+
require.NoError(t, NewCompiler().CompileWorkflow(workflowPath))
231+
lock, err := os.ReadFile(filepath.Join(dir, "harness.lock.yml"))
232+
require.NoError(t, err)
233+
const marker = "GH_AW_HARNESS_VERSION: "
234+
lineStart := bytes.Index(lock, []byte(marker))
235+
require.NotEqual(t, -1, lineStart, "compiled workflow should expose the experiment identity")
236+
lineEnd := bytes.IndexByte(lock[lineStart:], '\n')
237+
require.NotEqual(t, -1, lineEnd, "compiled identity should end at a newline")
238+
return strings.TrimSpace(string(lock[lineStart+len(marker) : lineStart+lineEnd]))
239+
}
240+
241+
assert.Equal(t, compile("First prompt body."), compile("Revised prompt body."))
242+
}
243+
180244
func TestGenerateExperimentSteps_SingleQuoteEscaping(t *testing.T) {
181245
c := &Compiler{}
182246
data := &WorkflowData{
@@ -691,17 +755,38 @@ func TestExperimentsBranchName(t *testing.T) {
691755

692756
// ── storage key is excluded from experiment configs ───────────────────────
693757

694-
func TestExtractExperimentConfigsFromFrontmatter_StorageKeyIsSkipped(t *testing.T) {
758+
func TestExtractExperimentConfigsFromFrontmatter_ReservedKeysAreSkipped(t *testing.T) {
695759
frontmatter := map[string]any{
696760
"experiments": map[string]any{
697-
"storage": "repo",
698-
"my_exp": []any{"A", "B"},
761+
"storage": "repo",
762+
"identity": "frontmatter",
763+
"my_exp": []any{"A", "B"},
699764
},
700765
}
701766
got := extractExperimentConfigsFromFrontmatter(frontmatter)
702767
require.NotNil(t, got, "experiment configs should not be nil")
703768
assert.Contains(t, got, "my_exp", "my_exp should be present")
704769
assert.NotContains(t, got, "storage", "storage key should be excluded from experiment configs")
770+
assert.NotContains(t, got, "identity", "identity key should be excluded from experiment configs")
771+
}
772+
773+
func TestExtractExperimentsIdentityFromFrontmatter(t *testing.T) {
774+
tests := []struct {
775+
name string
776+
frontmatter map[string]any
777+
want ExperimentIdentityMode
778+
}{
779+
{name: "defaults to frontmatter", frontmatter: map[string]any{}, want: ExperimentsIdentityFrontmatter},
780+
{name: "full", frontmatter: map[string]any{"experiments": map[string]any{"identity": "full"}}, want: ExperimentsIdentityFull},
781+
{name: "frontmatter", frontmatter: map[string]any{"experiments": map[string]any{"identity": "frontmatter"}}, want: ExperimentsIdentityFrontmatter},
782+
{name: "workflow", frontmatter: map[string]any{"experiments": map[string]any{"identity": "workflow"}}, want: ExperimentsIdentityWorkflow},
783+
{name: "unknown defaults to frontmatter", frontmatter: map[string]any{"experiments": map[string]any{"identity": "other"}}, want: ExperimentsIdentityFrontmatter},
784+
}
785+
for _, tt := range tests {
786+
t.Run(tt.name, func(t *testing.T) {
787+
assert.Equal(t, tt.want, extractExperimentsIdentityFromFrontmatter(tt.frontmatter))
788+
})
789+
}
705790
}
706791

707792
func TestParseExperimentMetricEvalReference(t *testing.T) {

‎pkg/workflow/compiler_orchestrator_workflow.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -647,6 +647,7 @@ func (c *Compiler) extractAdditionalConfigurations( //nolint:largefunc // Existi
647647
workflowData.ExperimentConfigs = extractExperimentConfigsFromFrontmatter(frontmatter)
648648
workflowData.Experiments = experimentVariantsFromConfigs(workflowData.ExperimentConfigs)
649649
workflowData.ExperimentsStorage = extractExperimentsStorageFromFrontmatter(frontmatter)
650+
workflowData.ExperimentsIdentity = extractExperimentsIdentityFromFrontmatter(frontmatter)
650651

651652
// Rewrite ${{ experiments.<name> }} references that ended up in engine.model (e.g.
652653
// `engine: { model: ${{ experiments.model }} }`) so the compiled workflow reads the

‎pkg/workflow/workflow_data.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -210,6 +210,7 @@ type WorkflowData struct {
210210
Experiments map[string][]string // A/B testing experiments: maps experiment name to variant list (from frontmatter)
211211
ExperimentConfigs map[string]*ExperimentConfig // Full A/B experiment metadata (populated alongside Experiments)
212212
ExperimentsStorage ExperimentStorageMode // "cache" or "repo" (default "repo"); controls how experiment state is persisted across runs
213+
ExperimentsIdentity ExperimentIdentityMode // "full", "frontmatter" (default), or "workflow"; controls experiment history identity across runs
213214
CachedConcurrencyGroupExprSet bool // true once CachedConcurrencyGroupExprErr has been populated; distinguishes "valid (nil)" from "not yet computed"
214215
CachedParsedToolsets []string // cached result of ParseGitHubToolsets for the GitHub tool (for performance optimization); populated by applyDefaults
215216
CachedAllowedDomainsStr string // cached allowed-domains string for sanitization (for performance optimization); computed once and reused across multiple compilation steps

‎specs/work-queue/HarnessIdentity.cfg‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,16 @@ SPECIFICATION Spec
22
CONSTANTS
33
FrontmatterHashes = {"fm-1", "fm-2"}
44
BodyHashes = {"body-1", "body-2"}
5+
FreshnessHashes = {"fresh-1", "fresh-2", "fresh-3"}
6+
WorkflowIDs = {"workflow-a"}
7+
IdentityModes = {"full", "frontmatter", "workflow"}
58
Revisions = {"rev-1", "rev-2", "rev-3"}
69
Workers = {"worker-1", "worker-2"}
710
None = "none"
811
INVARIANT TypeOK
9-
INVARIANT EvaluationIdentityIsFrontmatter
10-
PROPERTY BodyEditPreservesIdentity
11-
PROPERTY FrontmatterEditUpdatesIdentity
12+
INVARIANT EvaluationIdentityUsesConfiguredMode
13+
PROPERTY BodyEditIdentityBehavior
14+
PROPERTY FrontmatterEditIdentityBehavior
1215
PROPERTY DispatchCapturesCurrentRevision
1316
PROPERTY DispatchBindingsAreImmutable
1417
CHECK_DEADLOCK FALSE

0 commit comments

Comments
 (0)