This document provides a high-level overview of the architecture of the Inference Gateway. The gateway is designed to be modular and extensible, so new providers and routing strategies drop in without changing the request surface clients see.

A unified OpenAI-compatible request enters the gateway and passes through a middleware pipeline - optional OIDC authentication, optional guardrails, and the MCP tool-call loop - before model routing resolves the target model and the provider proxy dispatches it to whichever upstream provider serves it. A2A agents are clients too: the CLI delegates tasks to them, and they call the same OpenAI-compatible API themselves. With A2A_ENABLED=true the gateway is also an A2A server - it serves one merged agent card at GET /.well-known/agent-card.json and relays every A2A JSON-RPC call on POST /a2a to the agents listed in A2A_AGENTS.

The pipeline order matches the binary: cmd/gateway/main.go registers the auth, guardrails, and MCP middlewares in exactly this sequence before the /v1/* handlers. The gateway is stateless - replicas scale horizontally behind any load balancer, and per-request state (tool-call iteration, MCP context) lives in the request lifecycle, not the process. See Supported Providers for the full provider matrix: OpenAI, DeepSeek, Anthropic, Cohere, Groq, Cloudflare, Ollama, Ollama Cloud, Google, Mistral, MiniMax, Moonshot, Nvidia, llama.cpp, Z-AI, and ElevenLabs.

The Inference Gateway is built to run on Kubernetes. Traffic enters through the Kubernetes Gateway API - an Envoy Gateway data plane fronting a Service - and reaches a pool of stateless gateway pods, each fronting the same provider proxy. The operator provisions these Gateway API resources from a Gateway CR's gatewayAPI spec (the successor to Ingress); see the Kubernetes Operator guide. Telemetry is scraped on a dedicated metrics port via a ServiceMonitor, and providers stay external.

Pods are interchangeable. Add capacity with an HPA, remove pods with rolling updates. The Monitoring Stack here represents the ServiceMonitor + Prometheus + Grafana pipeline kube-prometheus-stack deploys around the gateway - see Observability for the full setup, and the Kubernetes Operator for managing this topology declaratively as Custom Resources.