Universal LLM Provider Runtime

One interface for every model. Authentication, routing, streaming, retries, caching — handled.

Quick Start · Features · Docs · Examples · Providers · Architecture · Contributing

What is mux

mux is the LLM provider runtime that powers the decoder coding agent. It handles everything between your application and LLM APIs — authentication, model resolution, streaming, retries, rate limiting, and caching.

When your app calls a model, mux figures out which provider to use, how to talk to it, and how to stream the response back. Switch from Anthropic to Ollama? mux handles the translation. API returns 529? mux retries with backoff. Response hits max_tokens? mux continues automatically.

Your app never talks to an LLM API directly. mux does.

Decoder is the product face: it owns UX, agent orchestration, tools, permissions, sessions, and product semantics. Mux is the provider engine: it owns credentials, catalog and route resolution, provider transports, normalized streams, retry/fallback, usage, and provider telemetry. Decoder integrates through the stable engine facade rather than assembling Mux's internal provider packages.

Ecosystem Boundaries

mux is a Decoder support engine. Keep the dependency edge one-way.

Hosts may import exactly four packages:

Everything else is engine-internal: provider, catalog, config, credentials, router, runtime, and their subpackages are not shared contracts. Enforced by decoder/scripts/check-mux-engine-boundary.sh and two Go AST tests in decoder/internal/testaudit/.

Exception: a fixed set of engine-internal symbols is reachable through the facade (for example credentials.Store behind engine.Options.SecretStore and operationsgraph.Input behind engine.OperationsGraphInput). Those symbols are frozen as part of the contract; see Frozen engine-internal types. engine/host_surface_test.go fails when that set changes.

• do not import decoder/internal/*
• do not import the removed legacy path decoder/shared/types

Quick Start

go get github.com/GrayCodeAI/mux@main

Until a post-rename release is tagged, use @main; the existing v0.0.1 tag declares a different Go module path.

Requires Go 1.27.2+ and a configured provider credential. Direct dependencies: UUID, tiktoken tokenizer, OS keyring, OpenTelemetry, pure-Go SQLite, and gRPC (linked only into -tags grpc builds of internal/grpc).

import (
    "context"
    "fmt"

    "github.com/GrayCodeAI/mux/engine"
)

// Hosts (like decoder) must use the stable engine facade
eng, err := engine.New(engine.Options{})
if err != nil { panic(err) }

sr, err := eng.Stream(context.Background(), engine.GenerateRequest{
    Messages: []engine.Message{{Role: "user", Content: "Hello"}},
    Requirements: engine.Requirements{Streaming: true},
    Preference: engine.Preference{
        PreferredProvider: "anthropic",
        PreferredModelID: "anthropic/claude-sonnet-4-6",
    },
})
if err != nil { panic(err) }
defer sr.Close()

for sr.Next() {
    if evt := sr.Event(); evt.Type == engine.EventContentDelta {
        fmt.Print(evt.Content)
    }
}
if err := sr.Err(); err != nil { panic(err) }

Provider construction lives under provider/; hosts use the stable engine contract. See docs/architecture/HOST-ENGINE-BOUNDARY.md and the feature-oriented architecture.

Features

Provider Routing

Automatically detects and routes to the right provider based on environment variables, config files, or explicit selection.

Model Resolution

Maps abstract tiers (opus/sonnet/haiku) to concrete model IDs per provider. Ships with an embedded catalog of pricing, context windows, and capabilities.

Streaming

Parses SSE for Anthropic and OpenAI formats — text, tool calls, and thinking blocks.

Reliability

• Retries on 429/500/529 with exponential backoff and Retry-After support
• Auto-continuation when stop_reason == max_tokens
• Provider fallback chains for high availability

Rate Limiting

Token bucket rate limiter per provider — prevents hitting API limits before they happen.

Caching

• Response caching with configurable TTL
• Semantic similarity caching for repeated prompts
• Anthropic prompt caching breakpoints on system prompt and conversation prefix

Cost Tracking

Built-in cost estimation per call, with per-provider pricing from the embedded model catalog.

Reasoning Controls

Passes reasoning_effort and Anthropic extended-thinking thinking_budget_tokens through to capable models — omitted when unset.

Keyless CI Auth

GitHub OIDC keyless authentication for cloud deployments — mints a short-lived token in GitHub Actions and exchanges it for AWS Bedrock (STS AssumeRoleWithWebIdentity) or GCP Vertex (Workload Identity Federation) credentials, no stored secrets.

OpenAI-Compatible Proxy

Serves POST /v1/chat/completions so existing OpenAI SDK clients can talk to mux unchanged.

Load-Balancing Strategies

Named routing strategies beyond weighted random: simple-shuffle, least-busy, latency-based, cost-based, and usage-based.

Pluggable Cache & Audit Sinks

Distributed CacheBackend interface (in-memory default, RESP/Redis-capable) and an AuditSink interface (no-op default, JSONL file sink) for privacy-preserving call metadata.

Model Role Slots

Named primary / weak / editor model slots with fallback to primary, plus an LLM summarizing condenser that shrinks long conversation histories using the weak model.

Rerank & Readiness

POST /rerank endpoint (provider-backed with lexical fallback) and a GET /ready readiness probe alongside the existing health check.

gRPC Transport (opt-in, internal)

internal/grpc holds an optional gRPC transport behind the grpc build tag. It serves mux.v1.ChatService/Chat with a registered json content subtype (no .proto files or generated stubs; clients call with grpc.CallContentSubtype("json")), backed by EngineChatService over conversation.Engine. The package is internal, so hosts cannot import it, and nothing in mux starts it. google.golang.org/grpc is a direct requirement in go.mod, so it appears in consumers' module graphs, but only -tags grpc builds link it.

Documentation

Detailed documentation is available in the docs/ directory:

• Architecture — System design, data flow, and reliability features
• Provider Setup Guide — Credential configuration and provider setup
• Dynamic Model Discovery — Live model discovery architecture

Examples

Runnable examples are in the examples/ directory:

• Basic Chat — Simple synchronous chat
• Streaming — SSE streaming with event handling
• Multi-Provider — Fallback chains across providers

Run any example with:

ANTHROPIC_API_KEY=sk-... go run ./examples/basic/

Supported Providers

28 provider gateways in catalog/registry/providers.go (decoder /config uses the same list), listed in registry SortOrder. catalog/registry/docs_test.go fails when this table, the count, or .env.example drift from the registry.

Runtime auto-detection uses a separate priority order (config.APIProviderDetectionOrder) when no deployment is pinned.

Usage

Basic Chat

resp, err := c.Chat(ctx, messages, llm.ChatOptions{
    Model: "gpt-4o",
})

Streaming with Continuation

// Auto-continues when max_tokens is hit
resp, err := provider.ChatWithContinuation(ctx, provider, messages,
    llm.ChatOptions{Model: model},
    core.DefaultContinuationConfig(),
)

Mock Provider for Testing

mock := provider.NewMockProvider(provider.MockModeFixed)
mock.Response = "Here is the code you asked for..."

resp, _ := mock.Chat(ctx, messages, opts)
// No real API calls — perfect for tests

Model Catalog

cat := catalog.DefaultModelCatalog()

// Get the best model for a tier
model := catalog.GetPreferredProviderModel("anthropic", catalog.TierSonnet, &cat)
// → "claude-sonnet-4-6"

// Check deprecation warnings
warn := catalog.GetModelDeprecationWarning("claude-3-7-sonnet", "anthropic")

Provider Configuration

cfg := config.LoadProviderConfig("")             // load from disk
config.ApplyProviderConfigToEnv(cfg, false, nil) // apply to environment
config.SaveProviderConfig(cfg, "")               // save changes

Architecture

mux/
├── engine/                 # Stable host-facing facade (hosts import engine, llm, graph, tools)
├── llm/                    # Host-facing DTOs and the Provider port that engine re-exports
├── graph/                  # Portable execution-graph vocabulary
├── tools/                  # Tool-call and tool-result contracts
├── provider/               # Provider runtime composition root (MuxClient)
│   ├── core/               # Provider-neutral wire, stream, retry, and transport primitives
│   ├── adapters/           # Provider protocol adapters and construction registry
│   ├── resilience/         # Rate limits, continuation, guardrails, and error policy
│   ├── cache/              # Response and semantic caches
│   ├── batch/              # Batch execution
│   ├── embeddings/         # Embedding clients, cache, and defaults
│   ├── media/              # Image and audio clients, structured prompts
│   ├── extraction/         # Structured extraction
│   ├── observability/      # Usage, cost, metrics, tracing, and recording
│   └── testkit/            # Mock provider for tests
├── catalog/                # Model catalog & tier system
│   ├── registry/           # Provider registry (single source of truth for providers)
│   ├── discover/           # Model discovery
│   ├── live/               # Live model listing per provider
│   ├── capabilities/       # Capability and deprecation data
│   └── concentrate/ opencodego/ opengateway/ xiaomi/ zai/  # Gateway-specific helpers
├── config/                 # Provider configuration & routing
│   └── credential/         # Credential file management
├── credentials/            # Keyring/env credential stores and OIDC keyless auth
├── router/                 # Routing strategies, deployment router, circuit breakers
│   └── controlplane/       # Versioned, signed peer manifests and replicas
├── runtime/                # Engine-internal provider/model/credential resolution
├── setup/                  # Catalog-backed deployment wiring
├── operationsgraph/        # Privacy-safe route and generation telemetry projection
├── conversation/           # Conversation engine with branching
├── storage/                # SQLite conversation DAG store, virtual keys, budgets
├── codeagent/              # Code agent retry & fallback strategies
├── verify/                 # Provider conformance harness
├── types/                  # Shared message types & API errors
├── constants/              # API limits
├── utils/                  # Error utilities
├── api/                    # OpenAPI spec for internal/api
├── internal/
│   ├── api/                # HTTP API server (library code; no mux binary starts it)
│   ├── cache/              # Cache backends and response cache warmer
│   ├── grpc/               # Optional gRPC transport (build tag grpc)
│   ├── health/             # Provider health checker
│   ├── httputil/ probehttp/ shrink/  # HTTP, probe, and tool-description helpers
│   ├── observability/      # OpenTelemetry spans, metrics, and audit sinks
│   └── sdk/                # Go, Python, TypeScript clients for the internal/api HTTP surface
├── docs/                   # Documentation & guides
├── examples/               # Runnable code examples
├── scripts/                # CI guards and helper scripts
└── assets/                 # Logo and branding

See docs/ARCHITECTURE.md for detailed system design and data flows.

operationsgraph.Build projects resolved routes and normalized usage into mux.graph/v1 operations nodes. Provider, model, request ID, and generated content are represented only by SHA-256 digests; token counts, finish reason, tool-call count, and deployment-routing state remain queryable.

Ecosystem

mux is part of the graycode-eco:

Development

Prerequisites

• Go 1.27.2+

Build & Test

go build ./...               # Verify the library compiles
go test -race ./...           # Run all tests with race detector
make ci                       # Run full CI suite (lint, test, security)
make cover                    # Generate coverage report

Contributing

We welcome contributions! Please see CONTRIBUTING.md for development setup, commit conventions, and the PR process.

Quick start:

1. Fork and create a branch: git checkout -b feat/short-description
2. Make changes in small, focused commits
3. Run make ci locally
4. Open a pull request

Use Conventional Commits for commit messages — release-please uses them for versioning.

License

MIT — see LICENSE for details.

© 2026 GrayCode AI

Project version

VERSION is the single version source. Edit it once, then run make version-sync to regenerate package and OpenAPI metadata. CI and release workflows run make version-check to reject stale metadata. Plain Go builds also carry the canonical version when run outside the checkout.