Repository navigation
Add experimental Agy engine and production conformance gate - #66903
Conversation
Prepare a manual-only, checksum-pinned native authentication probe before engine registration or Gemini deprecation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Implement the embedded native API-key profile, checksum-verified installation, proxy credential isolation, gateway MCP conversion and canonical/local session parsing. Reject unsupported restrictions and profile overrides; preserve Gemini execution and WIF. Add bounded read-only production conformance preparation and document the pending release gate. Gemini deprecation and release remain withheld until reviewed live conformance passes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff
|
|
✅ Ponytail Reviewer completed successfully!
|
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.
|
|
🔎 PR Code Quality Reviewer is reviewing code quality for this pull request... |
Visual Regression Check
Every navigation to Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "clients2.google.com"
- "host.docker.internal"See Network Configuration for more information.
|
✅ Design Decision Gate — ADR VerifiedADR reviewed: ADR-34693: Add Experimental Agy and Soft-Deprecate Gemini — implementation aligns with the stated decision. Great work! 🏗️ Gate triggered by code volume (843 additions in business-logic directories, threshold 100); the 📋 Verification SummaryChecked the ADR Decision commitments against the PR diff:
No divergences and no uncovered scope creep were found. Remaining non-blocking observation: the ADR records alternatives (rename-in-place, silent Gemini alias, dated removal, dedicated copied runtime, new provider target) inside Consequences rather than under an explicit 📋 What to do next
References:
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
One small simplification found.
net: -2 lines possible.
Generated by ✂️ Ponytail Reviewer for #66903 · codex · gpt56 · 22.9 AIC · ⌖ 6.33 AIC · ⊞ 13.4K
Comment /ponytail to run again
There was a problem hiding this comment.
Impeccable review — Agy experimental engine PR
Applied critique/audit lens (mixed/engineering-focused change: new experimental engine + conformance gate, no end-user UI surface) plus a correctness/security pass given the gh-aw review-config priorities (security > correctness > reliability > maintainability).
Findings: No blocking issues identified in the changed lines.
Strengths observed:
- Subprocess invocations (
scripts/agy-authentication.cjs, the embedded harness inpkg/workflow/data/engines/agy.md) use array-formspawn/spawnSync, avoiding shell-injection surfaces even with adversarial prompt content (verified by the "Literal $(touch SHOULD_NOT_EXIST)" harness test). - Native binary is checksum-verified (SHA-256) before execution, and executable
--versionoutput is re-checked against the pinned release. - Provider credentials stay outside the agent sandbox (dummy
GEMINI_API_KEY=awf-proxy+ AWF-discovered endpoint, no direct-inference fallback on discovery failure); evidence artifacts (report.json) are scrubbed to allowlisted fields only (tests assert secrets never appear inJSON.stringify(report)). - Fail-closed semantics are enforced thoroughly: pending tools, permission denials, interrupted runs, zero/invalid usage, and even a SIGINT race that tries to smuggle a stale "SUCCESS" result are all converted to
session.error(seeTestAgyHarnessFailsClosed/TestAgyHarnessWatchdogAndParentInterruption). - Backward compatibility for
geminiis preserved (dedicated tests assertGetGeminiAPITarget/DefaultGeminiAPITargetare unchanged), and the new engine is clearly marked experimental everywhere (CLI help, interactive menu, compiler diagnostic, docs). - Good breadth of test coverage across Go (compiler validation, harness, MCP adapter, API targets) and JS (log parser, unified session, authentication probe), including negative/adversarial cases.
Nothing here met the bar for a blocking inline comment. Given the PR is intentionally kept in draft pending production conformance evidence, this is a non-blocking COMMENT review.
🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 97.7 AIC · ⌖ 13.2 AIC · ⊞ 8.1K
There was a problem hiding this comment.
🟡 Changes recommended
Process cancellation, cumulative accounting, and conformance-gate correctness need fixes before approval.
5 open findings
What changed in this PR
Adds experimental Agy support to gh-aw without changing existing Gemini workflows or the default Copilot engine. Production conformance remains a release gate.
Changes:
- Registers Agy with pinned installation, isolated authentication, MCP configuration, and execution safeguards.
- Adds native event parsing and regression coverage.
- Prepares authentication and production conformance workflows, plus experimental setup documentation.
| File | Description |
|---|---|
| scripts/agy-authentication.test.cjs | Tests authentication probes and process bounds. |
| scripts/agy-authentication.cjs | Implements native authentication checks. |
| pkg/workflow/engine_catalog_test.go | Includes Agy in catalog expectations. |
| pkg/workflow/engine_api_targets.go | Routes Agy through Gemini endpoints. |
| pkg/workflow/data/engines/agy.md | Defines the experimental Agy runtime. |
| pkg/workflow/compiler_yaml_lookups.go | Resolves embedded engine versions. |
| pkg/workflow/compiler_orchestrator_tools.go | Rejects unsupported Agy tool restrictions. |
| pkg/workflow/compiler_orchestrator_engine.go | Applies Agy defaults and validation. |
| pkg/workflow/behavior_defined_engine.go | Isolates Gemini provider credentials. |
| pkg/workflow/agy_harness_test.go | Tests runtime safeguards and MCP conversion. |
| pkg/workflow/agy_engine_validation.go | Validates the supported Agy profile. |
| pkg/workflow/agy_engine_test.go | Tests registration, compilation, and conformance configuration. |
| pkg/workflow/agentic_engine.go | Registers embedded behavior-defined engines. |
| pkg/constants/engine_constants.go | Adds Agy identity and selection metadata. |
| pkg/constants/constants_test.go | Updates engine-list expectations. |
| pkg/cli/secrets_command_test.go | Checks Agy secret-bootstrap help. |
| pkg/cli/logs_filtering_test.go | Uses shared engine-help expectations. |
| pkg/cli/flags.go | Labels Agy experimental in help. |
| pkg/cli/completions_test.go | Updates engine completion count. |
| pkg/cli/cli_consistency_help_test.go | Aligns documentation checks with help. |
| pkg/cli/agy_session_parser_test.go | Tests local Agy session reconstruction. |
| Makefile | Adds authentication script tests. |
| docs/src/content/docs/setup/cli.md | Documents Agy CLI selection. |
| docs/src/content/docs/reference/engines.md | Explains Agy capabilities and release gates. |
| docs/src/content/docs/reference/auth.mdx | Documents API-key support and WIF limitations. |
| docs/src/content/docs/engines/agy.md | Adds experimental setup and migration guidance. |
| docs/astro.config.mjs | Adds Agy navigation. |
| docs/adr/34693-add-antigravity-engine-deprecate-gemini.md | Revises the draft integration decision. |
| actions/setup/session_parsers.go | Embeds the Agy parser. |
| actions/setup/js/unified_session.cjs | Routes Agy session parsing. |
| actions/setup/js/parse_custom_log.cjs | Detects native Agy events. |
| actions/setup/js/parse_agy_log.test.cjs | Tests normalization and cumulative accounting. |
| actions/setup/js/parse_agy_log.cjs | Normalizes native Agy logs. |
| .github/workflows/engine-conformance-agy.md | Defines the production conformance gate. |
| .github/workflows/credentials-check.yml | Adds opt-in Agy validation jobs. |
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /codebase-design (architecture_change classification). This is a large, carefully-scoped PR with strong test coverage (engine registration, harness fail-closed paths, MCP adapter, conformance gate bounds) and an explicit draft-until-conformance-passes gate — the engineering discipline here is notable.
📋 Key Themes & Highlights
Key Themes
- Validation gap:
validateAgyEngineConfigrejects every override surface (args,harness,driver,cwd,config,max-turns, ...) exceptengine.command, which the docs mention as supported but the validator never checks — an inconsistency given the function's otherwise "fail loud" design. - Switch-case scalability: the new
*BehaviorDefinedEnginebranch ingetEngineAPIHostsreturnsnilhosts for any non-Gemini-secret-strategy behavior-defined engine, which is correct today (onlyagyqualifies) but silently drops firewall host inference for the next declarative engine added without a reviewer noticing the implicit coupling. - Harness navigability: the ~85-line native harness and MCP config-adapter are embedded as YAML string literals in
agy.mdrather than standalone.cjsfiles, making the highest-risk code (credential isolation, checksum verification, process lifecycle) the one piece without direct JS linting/test-runner coverage.
Positive Highlights
- ✅ Fail-closed harness design (
agy_harness_test.go) covers denial, interruption, watchdog escalation, and malformed-stream cases thoroughly. - ✅ Clear separation of Gemini vs. Agy identity (distinct engine ID, no aliasing, no silent Gemini behavior change) — the ADR revision addresses prior design concerns well.
- ✅ The production conformance gate is appropriately bounded (10-minute timeout, no write permissions, staged safe-outputs) and tested for those bounds (
TestAgyProductionConformanceIsBoundedAndReadOnly).
None of the three findings above are blocking correctness bugs — they're consistency/navigability gaps worth tightening before Gemini soft-deprecation work builds on this foundation.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 182.6 AIC · ⌖ 14.9 AIC · ⊞ 10.1K
Comment /matt to run again
Keep the engine catalog dispatch-only and use a separate reusable gate for credentials checks. Share the native probes and prompt while preserving bounded read-only conformance and covering both compilation paths. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Visual Regression Check: could not completeThe docs server at Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "clients2.google.com"
- "host.docker.internal"See Network Configuration for more information.
|
Bound process-group cancellation and inherited-pipe cleanup, preserve cumulative native metrics, and require staged noop evidence from the real safe-output handler. Disable unrelated Copilot detection in both Gemini-only conformance entrypoints and cover trusted custom executables explicitly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Regenerate conflicted engine conformance locks from merged sources while preserving experimental Agy runtime and staged evidence checks. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Include upstream smoke model-expression and debugging guidance fixes through main commit 697ea9a, with generated workflows rebuilt from the merged compiler. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@copilot address the following outstanding work in one pass:
Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI. Sous-chef head: 00e8f4d
|
The branch already contains the latest |
Visual Regression CheckAll 3 viewports (mobile 375, tablet 768, desktop 1440) of Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "clients2.google.com"
- "host.docker.internal"See Network Configuration for more information.
|
|
🎉 This pull request is included in a new release. Release: |
Why
Add the native Google Antigravity CLI as an explicitly experimental built-in engine without silently changing existing Gemini workflows. This is an authentication-first implementation checkpoint for the planned Agy integration and Gemini soft deprecation, not a release-ready replacement.
Related to #66713. This draft does not close that issue.
Approach
agythrough an embedded declarative engine definition. Short-form and object-form selection, CLI overrides, interactive metadata and default version/model resolution require no custom imports. Copilot remains the default.modelProvider: geminisettings. Keep the provider key outside the agent sandbox and discover the existing Gemini AWF endpoint without direct-inference fallback.GEMINI_API_KEY; generated jobs have no GitHub write permissions and use staged safe outputs.Compatibility and release gate
Existing Gemini binaries, version pins, API-key/WIF authentication, generated conformance workflow and historical parser remain unchanged. No alias, automatic migration, removal date or Gemini deprecation notice is introduced yet. Agy ADC/WIF remains a separately gated follow-up; WIF users must retain
engine: gemini.Keep this PR in draft until production conformance passes. Native authentication passed on a clean Actions runner in run 37732637929, including fresh inference, missing/invalid credentials, an unknown model and controlled endpoint routing. That does not prove production AWF, native MCP or safe-output execution. The production workflow is prepared but has not been dispatched. Native tool-event assumptions still require real-run evidence. Gemini soft deprecation and the combined release changeset are deliberately withheld until that gate passes.
Initial implementation validation
make fmt,make build, andmake recompilepassed; the original 328 workflow locks were in sync.make agent-report-progresspassed formatting, lint, schema freshness and its selected impacted tests../gh-aw compile engine-conformance-agy --dry-run --action-mode dev --no-check-update --jsonpassed strict compilation and shellcheck with no warnings for this workflow.GOOS=js GOARCH=wasm go build ./pkg/workflowpassed. Existing Gemini runtime, version constants, parser and conformance sources/lock were verified unchanged.Docker-based scanner and production runtime evidence are not claimed.
PR check fix: cf4c19c
impacted-go-testswas PR-caused:TestEngineConformanceCatalogCoverage/agyrejected the canonical workflow's extraworkflow_calltrigger atpkg/workflow/engine_conformance_test.go:191. The dispatch-only catalog invariant is unchanged.engine-conformance-agy.mdnow remains dispatch-only; Credentials Check calls the feature-branchagy-conformance-reusable.lock.yml. Directly importing the canonical workflow was rejected by the existing shared-import trigger validation, so both entry points import one copy of the native probes and prompt fromshared/agy-conformance.md.The native MCP configuration, challenge/checker scripts, evidence upload and prompt were verified byte-for-byte unchanged during extraction. Both generated paths retain shared fresh nonce/math/file/shell-environment/CLI MCP probes, native MCP transport, requested-model and positive inference-usage checks, staged noop, five-credit/twelve-cache-miss budgets, ten-minute agent/job and execution-step limits, two-minute safe-output limits, threat detection, and no GitHub write permissions. The caller still grants only
actions: read/contents: readand forwards onlyGEMINI_API_KEY.Local validation for the fix:
./gh-aw compile engine-conformance-agy agy-conformance-reusable --validate --strict: 2 succeeded, 0 warnings on the final focused compilation.go test ./pkg/workflow -run '^(TestEngineConformance|TestAgy)' -count=1 -v: passed, including both compiled entry points, shared probe execution/negative cases and the unchanged catalog invariant (8.782s on the final run).make fmt: passed.make recompile: 329/329 succeeded; existing repository warnings remain.make agent-report-progress: passed Go/custom lint, JavaScript lint, schema checks, selected impacted Go tests, 10 selected JavaScript tests, and full workflow drift validation. An initial new-test testifylint failure was corrected; task-created compiler/cache directories were cleaned before the successful final run.These are local results, not a claim that hosted checks or live production conformance passed. No live conformance was dispatched; Gemini deprecation and release status remain unchanged.
JS Tests (shard 1/4)is pre-existing and is not fixed here. The prior investigation established that the routing test/runtime/helper blobs match PR basea4ca9f228c; four endpoint fixtures omitprovider. The failing file isactions/setup/js/copilot_harness.test.cjs, suitecopilot_harness.cjs > AWF task-level model routing:applies the selected model, endpoint and effort over existing values(assertion line 169)accepts router-selected reasoning effort none(line 193)accepts router-selected reasoning effort max(line 193)rejects unknown router-selected reasoning efforts(line 214)Hosted job 113362498542 reports 4 failed, 4625 passed, 5 skipped; the prior targeted reproduction reported 4 failed, 4 passed, 322 deselected. The final changed-file JS selection did not include these fixtures. They require a separate follow-up, not weakened assertions or an unrelated fix in this PR.
Reusable-wrapper security review
The new generated wrapper adds a reference to the existing
GEMINI_API_KEY, not a new repository credential or inference endpoint. Its manifest carries the same existing optional token/telemetry references (COPILOT_GITHUB_TOKEN,GH_AW_DEFAULT_OTLP_ENDPOINT,GH_AW_DEFAULT_OTLP_HEADERS,GH_AW_GITHUB_MCP_SERVER_TOKEN,GH_AW_GITHUB_TOKEN, andGITHUB_TOKEN); Credentials Check forwards none of these optional secrets. The actual Gemini key remains excluded from the agent environment by AWF. No secrets were removed, and no action/container pins, dependencies or endpoint redirects were added or changed: the wrapper reuses the canonical gate's reviewed configuration. Native authentication remains distinct from the still-pending live production gate.