Skip to content

Commit cf4c19c

Browse files
pelikhanCopilot
pelikhan
and
Copilot
committed
Fix Agy conformance workflow entry points
Keep the engine catalog dispatch-only and use a separate reusable gate for credentials checks. Share the native probes and prompt while preserving bounded read-only conformance and covering both compilation paths. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent 4ade037 commit cf4c19c

10 files changed

Lines changed: 2590 additions & 238 deletions

‎.github/workflows/agy-conformance-reusable.lock.yml‎

Lines changed: 2274 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.github/workflows/agy-conformance-reusable.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
name: Agy Conformance Reusable (Experimental)
3+
description: Feature-branch reusable entry point for the canonical experimental Agy conformance gate.
4+
on:
5+
workflow_call:
6+
permissions:
7+
contents: read
8+
concurrency:
9+
job-discriminator: ${{ github.run_id }}
10+
strict: true
11+
inlined-imports: true
12+
timeout-minutes: 10
13+
jobs:
14+
agent:
15+
timeout-minutes: 10
16+
max-ai-credits: 5
17+
features:
18+
gh-aw-detection: false
19+
engine:
20+
id: agy
21+
env:
22+
ENGINE_CONFORMANCE_SENTINEL: conformance-agy
23+
safe-outputs:
24+
staged: true
25+
activation-comments: false
26+
report-failure-as-issue: false
27+
report-failed-jobs: false
28+
timeout-minutes: 2
29+
noop:
30+
report-as-issue: false
31+
missing-tool:
32+
create-issue: false
33+
imports:
34+
- shared/agy-conformance.md
35+
---
36+
37+
Execute the imported Agy conformance gate once.

‎.github/workflows/credentials-check.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,6 @@ jobs:
159159
permissions:
160160
actions: read
161161
contents: read
162-
uses: ./.github/workflows/engine-conformance-agy.lock.yml
162+
uses: ./.github/workflows/agy-conformance-reusable.lock.yml
163163
secrets:
164164
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}

‎.github/workflows/engine-conformance-agy.lock.yml‎

Lines changed: 34 additions & 123 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.github/workflows/engine-conformance-agy.md‎

Lines changed: 2 additions & 87 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@ name: Engine Conformance Agy (Experimental)
33
description: Bounded production authentication, AWF, MCP and safe-output gate for experimental Agy.
44
on:
55
workflow_dispatch:
6-
workflow_call:
76
permissions:
87
contents: read
98
concurrency:
@@ -15,7 +14,6 @@ jobs:
1514
agent:
1615
timeout-minutes: 10
1716
max-ai-credits: 5
18-
max-turn-cache-misses: 12
1917
features:
2018
gh-aw-detection: false
2119
engine:
@@ -32,91 +30,8 @@ safe-outputs:
3230
report-as-issue: false
3331
missing-tool:
3432
create-issue: false
35-
mcp-servers:
36-
agy-native:
37-
type: http
38-
url: "http://host.docker.internal:${{ steps.mcp-scripts-start.outputs.port }}"
39-
headers:
40-
Authorization: "${{ steps.mcp-scripts-start.outputs.api_key }}"
41-
allowed:
42-
- native-challenge
43-
mcp-scripts:
44-
native-challenge:
45-
description: Return a fresh nonce and record evidence for the native Agy MCP transport.
46-
inputs:
47-
file_nonce:
48-
type: string
49-
required: true
50-
description: The fileNonce from the conformance fixture.
51-
env:
52-
CONFORMANCE_STATE: ${{ runner.temp }}/engine-conformance
53-
script: |
54-
const crypto = require("node:crypto");
55-
const fs = require("node:fs");
56-
const path = require("node:path");
57-
const host = process.env.CONFORMANCE_STATE;
58-
if (!host) throw new Error("CONFORMANCE_STATE is missing");
59-
const expected = JSON.parse(fs.readFileSync(path.join(host, "expected.json"), "utf8"));
60-
if (file_nonce !== expected.fileNonce) throw new Error("The fixture nonce does not match");
61-
const receipt = { fileNonce: file_nonce, toolNonce: crypto.randomBytes(24).toString("hex") };
62-
fs.writeFileSync(path.join(host, "native-receipt.json"), JSON.stringify(receipt), { mode: 0o600 });
63-
return { toolNonce: receipt.toolNonce };
64-
post-steps:
65-
- name: Assert native Agy MCP transport and inference accounting
66-
if: always()
67-
env:
68-
CONFORMANCE_STATE: ${{ runner.temp }}/engine-conformance
69-
run: |
70-
set -euo pipefail
71-
node <<'JS'
72-
const assert = require("node:assert/strict");
73-
const fs = require("node:fs");
74-
const path = require("node:path");
75-
const host = process.env.CONFORMANCE_STATE;
76-
const receipt = JSON.parse(fs.readFileSync(path.join(host, "native-receipt.json"), "utf8"));
77-
const expected = JSON.parse(fs.readFileSync(path.join(host, "expected.json"), "utf8"));
78-
assert.equal(receipt.fileNonce, expected.fileNonce);
79-
assert.match(receipt.toolNonce, /^[a-f0-9]{48}$/);
80-
const entries = fs.readFileSync("/tmp/gh-aw/agent-stdio.log", "utf8").split(/\r?\n/).flatMap(line => {
81-
try { return [JSON.parse(line)]; } catch { return []; }
82-
});
83-
const tools = entries.filter(entry => entry.event === "step_update" && entry.step_update?.step_type === "tool");
84-
assert.ok(tools.some(entry => /agy[-_]native.*native[-_]challenge/.test(entry.step_update.tool_info?.name || "")),
85-
"The challenge must be called through the native MCP client, not its CLI wrapper");
86-
const terminal = entries.filter(entry => entry.event === "result").at(-1)?.result;
87-
assert.equal(terminal?.status, "SUCCESS");
88-
assert.ok(Number.isSafeInteger(terminal?.num_turns) && terminal.num_turns > 0);
89-
assert.ok(terminal?.usage?.input_tokens > 0);
90-
assert.ok(terminal?.usage?.output_tokens > 0);
91-
assert.ok(entries.some(entry => entry.event === "init" && entry.init?.model === "gemini-3.8-flash-medium"));
92-
const usage = {};
93-
for (const name of ["input_tokens", "output_tokens", "thinking_tokens", "cache_read_tokens", "total_tokens"]) {
94-
const count = terminal.usage[name];
95-
if (count !== undefined) {
96-
assert.ok(Number.isSafeInteger(count) && count >= 0, `Invalid ${name}`);
97-
usage[name] = count;
98-
}
99-
}
100-
fs.writeFileSync(path.join(host, "native-report.json"),
101-
JSON.stringify({ status: "passed", nativeMCP: true, inference: true, usage }, null, 2),
102-
{ mode: 0o600 });
103-
JS
104-
- name: Upload native Agy conformance evidence
105-
if: always()
106-
uses: actions/upload-artifact@v7.0.2
107-
with:
108-
name: agy-production-native-conformance
109-
path: ${{ runner.temp }}/engine-conformance/native-report.json
110-
retention-days: 2
111-
if-no-files-found: error
11233
imports:
113-
- uses: shared/engine-conformance.md
114-
with:
115-
engine-id: agy
34+
- shared/agy-conformance.md
11635
---
11736

118-
Execute the imported engine configuration conformance suite once.
119-
120-
Also call the `native-challenge` tool through the native MCP server `agy-native`,
121-
using the actual fixture `fileNonce`. Do not call `mcpscripts native-challenge`
122-
or substitute a shell/HTTP request; the checker requires a native MCP tool event.
37+
Execute the imported Agy conformance gate once.

‎.github/workflows/shared/agy-conformance.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
---
2+
max-turn-cache-misses: 12
3+
mcp-servers:
4+
agy-native:
5+
type: http
6+
url: "http://host.docker.internal:${{ steps.mcp-scripts-start.outputs.port }}"
7+
headers:
8+
Authorization: "${{ steps.mcp-scripts-start.outputs.api_key }}"
9+
allowed:
10+
- native-challenge
11+
mcp-scripts:
12+
native-challenge:
13+
description: Return a fresh nonce and record evidence for the native Agy MCP transport.
14+
inputs:
15+
file_nonce:
16+
type: string
17+
required: true
18+
description: The fileNonce from the conformance fixture.
19+
env:
20+
CONFORMANCE_STATE: ${{ runner.temp }}/engine-conformance
21+
script: |
22+
const crypto = require("node:crypto");
23+
const fs = require("node:fs");
24+
const path = require("node:path");
25+
const host = process.env.CONFORMANCE_STATE;
26+
if (!host) throw new Error("CONFORMANCE_STATE is missing");
27+
const expected = JSON.parse(fs.readFileSync(path.join(host, "expected.json"), "utf8"));
28+
if (file_nonce !== expected.fileNonce) throw new Error("The fixture nonce does not match");
29+
const receipt = { fileNonce: file_nonce, toolNonce: crypto.randomBytes(24).toString("hex") };
30+
fs.writeFileSync(path.join(host, "native-receipt.json"), JSON.stringify(receipt), { mode: 0o600 });
31+
return { toolNonce: receipt.toolNonce };
32+
post-steps:
33+
- name: Assert native Agy MCP transport and inference accounting
34+
if: always()
35+
env:
36+
CONFORMANCE_STATE: ${{ runner.temp }}/engine-conformance
37+
run: |
38+
set -euo pipefail
39+
node <<'JS'
40+
const assert = require("node:assert/strict");
41+
const fs = require("node:fs");
42+
const path = require("node:path");
43+
const host = process.env.CONFORMANCE_STATE;
44+
const receipt = JSON.parse(fs.readFileSync(path.join(host, "native-receipt.json"), "utf8"));
45+
const expected = JSON.parse(fs.readFileSync(path.join(host, "expected.json"), "utf8"));
46+
assert.equal(receipt.fileNonce, expected.fileNonce);
47+
assert.match(receipt.toolNonce, /^[a-f0-9]{48}$/);
48+
const entries = fs.readFileSync("/tmp/gh-aw/agent-stdio.log", "utf8").split(/\r?\n/).flatMap(line => {
49+
try { return [JSON.parse(line)]; } catch { return []; }
50+
});
51+
const tools = entries.filter(entry => entry.event === "step_update" && entry.step_update?.step_type === "tool");
52+
assert.ok(tools.some(entry => /agy[-_]native.*native[-_]challenge/.test(entry.step_update.tool_info?.name || "")),
53+
"The challenge must be called through the native MCP client, not its CLI wrapper");
54+
const terminal = entries.filter(entry => entry.event === "result").at(-1)?.result;
55+
assert.equal(terminal?.status, "SUCCESS");
56+
assert.ok(Number.isSafeInteger(terminal?.num_turns) && terminal.num_turns > 0);
57+
assert.ok(terminal?.usage?.input_tokens > 0);
58+
assert.ok(terminal?.usage?.output_tokens > 0);
59+
assert.ok(entries.some(entry => entry.event === "init" && entry.init?.model === "gemini-3.8-flash-medium"));
60+
const usage = {};
61+
for (const name of ["input_tokens", "output_tokens", "thinking_tokens", "cache_read_tokens", "total_tokens"]) {
62+
const count = terminal.usage[name];
63+
if (count !== undefined) {
64+
assert.ok(Number.isSafeInteger(count) && count >= 0, `Invalid ${name}`);
65+
usage[name] = count;
66+
}
67+
}
68+
fs.writeFileSync(path.join(host, "native-report.json"),
69+
JSON.stringify({ status: "passed", nativeMCP: true, inference: true, usage }, null, 2),
70+
{ mode: 0o600 });
71+
JS
72+
- name: Upload native Agy conformance evidence
73+
if: always()
74+
uses: actions/upload-artifact@v7.0.2
75+
with:
76+
name: agy-production-native-conformance
77+
path: ${{ runner.temp }}/engine-conformance/native-report.json
78+
retention-days: 2
79+
if-no-files-found: error
80+
imports:
81+
- uses: shared/engine-conformance.md
82+
with:
83+
engine-id: agy
84+
---
85+
86+
Execute the imported engine configuration conformance suite once.
87+
88+
Also call the `native-challenge` tool through the native MCP server `agy-native`,
89+
using the actual fixture `fileNonce`. Do not call `mcpscripts native-challenge`
90+
or substitute a shell/HTTP request; the checker requires a native MCP tool event.

‎docs/adr/34693-add-antigravity-engine-deprecate-gemini.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,11 @@ model and controlled endpoint routing. That evidence does not prove AWF or MCP.
8686
Release of Agy and Gemini soft deprecation together requires bounded production
8787
conformance through the actual installer, Gemini AWF endpoint, native MCP,
8888
CLI-mounted MCP and staged safe outputs. The manual
89-
`engine-conformance-agy.md` workflow prepares that gate. Production conformance
89+
`engine-conformance-agy.md` workflow prepares that gate and remains dispatch-only.
90+
Credentials Check calls the feature-branch `agy-conformance-reusable.lock.yml`.
91+
Both entry points import `shared/agy-conformance.md`, preserving one copy of the
92+
native/shared probes and prompt without importing a trigger-bearing workflow.
93+
Production conformance
9094
has not passed; no Gemini deprecation notice or release changeset is enabled by
9195
this implementation checkpoint.
9296

‎docs/src/content/docs/engines/agy.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,9 @@ credential inside the agent.
8989
The manual native authentication probe checks real inference and negative
9090
authentication/model cases. `engine-conformance-agy.md` separately exercises the
9191
production installer, AWF, native MCP, CLI-mounted MCP tools and staged safe
92-
outputs. Release and Gemini deprecation are gated on that production path, not
92+
outputs. It remains dispatch-only; Credentials Check calls the feature-branch
93+
`agy-conformance-reusable.lock.yml`, compiled from the same shared probes and
94+
prompt. Release and Gemini deprecation are gated on that production path, not
9395
on mocked tests or native authentication alone.
9496

9597
See the [engine reference](/gh-aw/reference/engines/) and

‎docs/src/content/docs/reference/engines.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -74,8 +74,11 @@ production-engine conformance test. The separate built-in `engine: agy`
7474
integration remains experimental. Its production gate additionally requires
7575
native MCP transport, inference usage and requested-model evidence through AWF.
7676

77-
The `agy-conformance` input calls `engine-conformance-agy.lock.yml` from the
78-
selected branch; it does not run the legacy credential jobs. Both Agy selectors
77+
The `agy-conformance` input calls `agy-conformance-reusable.lock.yml` from the
78+
selected branch; it does not run the legacy credential jobs. The canonical
79+
`engine-conformance-agy.md` remains dispatch-only. Both entry points import the
80+
same probes and prompt from `shared/agy-conformance.md`; trigger-bearing
81+
workflows cannot be imported as shared Markdown. Both Agy selectors
7982
have isolated, ref-specific concurrency. The production gate has a ten-minute
8083
job timeout, five-credit budget, and staged safe outputs. Native authentication
8184
has passed; production conformance and Gemini soft deprecation remain pending.

0 commit comments

Comments
 (0)