Skip to content

bug: gemini API key rejected by proxy sidecar despite valid key #25944

Closed
Closed
bug: gemini API key rejected by proxy sidecar despite valid key#25944
Labels
ai-inspectedawfcommunity

Description

@duncankmckinnon
duncankmckinnon
opened on Apr 12, 2026

engine: gemini — API key rejected by proxy sidecar despite valid key

When using engine: gemini in an agentic workflow, the Gemini API returns API_KEY_INVALID even though the key is valid and confirmed working via direct curl requests.

Evidence

  • The validation step passes: ✅ GEMINI_API_KEY: Configured
  • Direct curl to generativelanguage.googleapis.com with the same key succeeds and returns model data
  • The workflow fails immediately with: 
    API key not valid. Please pass a valid API key.
status: INVALID_ARGUMENT

Proxy configuration from logs

[INFO] API proxy enabled: OpenAI=false, Anthropic=false, Copilot=false, Gemini=true

The lock file routes Gemini calls through the proxy sidecar:

GEMINI_API_BASE_URL: http://host.docker.internal:10003
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}

Suspected cause

The API proxy sidecar at host.docker.internal:10003 appears to not be forwarding the API key correctly to generativelanguage.googleapis.com, or is stripping/modifying it in transit.

Reproduction

  1. Create a workflow with engine: gemini
  2. Add a valid GEMINI_API_KEY as a repo secret
  3. Run the workflow — validation passes but agent execution fails with API_KEY_INVALID

Environment

  • gh-aw compiler: v0.68.1
  • Gemini CLI: bundled version from node/24.14.1
  • Runner: ubuntu-latest

Activity

  1. duncankmckinnon
    changed the title [-]engine: gemini — API key rejected by proxy sidecar despite valid key[/-] [+]bug: gemini API key rejected by proxy sidecar despite valid key[/+] on Apr 12, 2026
  2. github-actions
    locked and limited conversation to collaborators on Apr 12, 2026
  3. github-actions
    unlocked this conversation on Apr 12, 2026
  4. github-actions
    added
    ai-inspected
    on Apr 12, 2026
  5. pelikhan
    added
    community
    on Apr 13, 2026

  6. dbrattli commented on Apr 21, 2026

    @dbrattli
    dbrattli
    on Apr 21, 2026

    Confirming this is still broken on v0.69.0 / firewall image 0.25.25.

    What we've verified:

    • GEMINI_API_KEY secret is correctly set and confirmed valid — direct curl to generativelanguage.googleapis.com with the key succeeds
    • Upgrading through v0.68.1 → v0.68.3 → v0.68.7 → v0.69.0 and recompiling each time does not fix it
    • The proxy now correctly reports Gemini=true (fixed in v0.68.3), but the key is still rejected by Google with API_KEY_INVALID

    What we see in logs:

    [INFO] API proxy enabled: OpenAI=false, Anthropic=false, Copilot=false, Gemini=true
[INFO] API proxy sidecar enabled - API keys will be held securely in sidecar container
...
_ApiError: {"error":{"code":400,"message":"API key not valid. Please pass a valid API key.","status":"INVALID_ARGUMENT","reason":"API_KEY_INVALID"}}

    Our interpretation: the proxy sidecar receives the request but forwards a placeholder key to Google rather than substituting the real GEMINI_API_KEY. The fix in gh-aw-firewall#1944 resolved the routing half (CLI now reaches Google) but the key injection half is still not working.

  7. github-actions
    locked and limited conversation to collaborators on Apr 21, 2026
  8. github-actions
    unlocked this conversation on Apr 21, 2026

  9. dbrattli commented on Apr 22, 2026

    @dbrattli
    dbrattli
    on Apr 22, 2026

    Update: still broken on firewall image 0.25.26

    PR gh-aw-firewall#1995 was merged on 2026-04-15 and shipped in v0.69.3 (firewall image 0.25.26). We upgraded and just ran the workflow — same result.

    Run: https://github.com/cognitedata/cognite-function-apps/actions/runs/24777513108/job/72499620442

    Log confirms:

    • API proxy enabled: Gemini=true
    • GEMINI_API_BASE_URL: http://host.docker.internal:10003
    • API proxy sidecar enabled - API keys will be held securely in sidecar container
    • --image-tag 0.25.26

    But still:

    _ApiError: {"error":{"code":400,"message":"API key not valid. Please pass a valid API key.","status":"INVALID_ARGUMENT","reason":"API_KEY_INVALID"}}

    The fix in gh-aw-firewall#1995 addresses stripping the placeholder header/query-param and injecting the real key — but the proxy sidecar may not be receiving GEMINI_API_KEY in its environment at all. The in-container pre-flight health check only checks ANTHROPIC_BASE_URL, OPENAI_BASE_URL, and COPILOT_API_URL — there's no equivalent Gemini health check, so failures are silent until the first real API call.

  10. github-actions
    locked and limited conversation to collaborators on Apr 22, 2026
  11. github-actions
    unlocked this conversation on Apr 22, 2026
  12. pelikhan
    added
    awf
    on Apr 23, 2026

  13. lpcox commented on Apr 23, 2026

    @lpcox
    lpcox
    on Apr 23, 2026
    Collaborator

    🔗 AWF tracking issue: https://github.com/github/gh-aw-firewall/issues/github/gh-aw-firewall#2173

    Generated by Firewall Issue Dispatcher · ● 1.2M · ◷

  14. lpcox commented on Apr 29, 2026

    @lpcox
    lpcox
    on Apr 29, 2026
    Collaborator

    🔗 AWF tracking issue: https://github.com/github/gh-aw-firewall/issues/github/gh-aw-firewall#2291

    Generated by Firewall Issue Dispatcher · ● 436.3K · ◷

  15. lpcox commented on Apr 29, 2026

    @lpcox
    lpcox
    on Apr 29, 2026
    Collaborator

    The Gemini API key forwarding issue was addressed in gh-aw-firewall by:

    • PR #2182 (merged) — strips all Gemini auth query param variants to prevent API_KEY_INVALID from double-injection
    • PR #2200 (merged) — added startup API key validation to catch invalid keys early

    Included in firewall v0.25.29+.

  16. lpcox
    closed this as completedon Apr 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ai-inspectedawfcommunity

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions