litellm-langfuse-caddy

git clone https://github.com/aaronmorris-dev/litellm-langfuse-caddy.git
cd litellm-langfuse-caddy

cp .env.example .env
cp litellm/config.example.yaml litellm/config.yaml

# Generate secrets and edit .env:
openssl rand -hex 32   # → SALT, ENCRYPTION_KEY, NEXTAUTH_SECRET
openssl rand -hex 16   # → LITELLM_MASTER_KEY

# Uncomment your providers in litellm/config.yaml, then:
chmod +x scripts/*.sh
./scripts/start.sh

After ./scripts/start.sh completes:

flowchart TD
    CLI["🖥️ CLI Tools
Claude Code · Gemini CLI · OpenAI SDK"]

    CLI -->|":4000 · :5002"| Caddy

    subgraph compose[" Docker Compose Stack "]
        Caddy["Caddy
Reverse Proxy · SSE"]

        Caddy --> LiteLLM["LiteLLM
Unified LLM Proxy"]
        Caddy --> Langfuse["Langfuse
Observability"]

        LiteLLM -.->|"traces"| Langfuse
        Worker["⚙️ Langfuse Worker"]

        LiteLLM --> PG & Redis
        Langfuse & Worker --> PG & CH & Redis & MinIO

        PG[("PostgreSQL")]
        CH[("ClickHouse")]
        Redis[("Redis")]
        MinIO[("MinIO")]
        PG ~~~ CH ~~~ Redis ~~~ MinIO
    end

    LiteLLM --> Providers["Bedrock · Vertex AI · Gemini · OpenAI · Anthropic"]

• Multi-provider routing — Bedrock, Vertex AI, Gemini, OpenAI, and Anthropic behind one endpoint. Switch models without changing your code.
• Per-tool tracking — Give each tool its own virtual key. Every request is automatically tagged to its source in Langfuse.
• Automatic trace enrichment — langfuse_enrich.py maps virtual keys to trace names, daily sessions, and tags. No client changes needed.
• Session evaluation — An LLM-as-judge script scores sessions on task completion, approach, and communication.
• Zero-buffered streaming — Caddy delivers tokens instantly with 10-minute response timeouts.
• Hardened by default — Memory limits, health checks, log rotation, and privilege lockdown on every service.

# 1. Configure SSO profile
aws configure sso --profile your-profile-name

# 2. Login
aws sso login --profile your-profile-name

AWS_PROFILE=your-profile-name

- ~/.aws:/root/.aws # writable — SSO needs token cache

- model_name: claude-sonnet-4-6
  litellm_params:
    model: bedrock/us.anthropic.claude-sonnet-4-6

gcloud auth application-default login

GOOGLE_APPLICATION_CREDENTIALS=/root/.config/gcloud/application_default_credentials.json

- ~/.config/gcloud:/root/.config/gcloud:ro

- model_name: gemini-2.5-flash
  litellm_params:
    model: vertex_ai/gemini-2.5-flash
    vertex_project: your-gcp-project
    vertex_location: us-central1

GEMINI_API_KEY=your-key

- model_name: gemini-2.5-flash
  litellm_params:
    model: gemini/gemini-2.5-flash

- OPENAI_API_KEY=${OPENAI_API_KEY}
# or
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}

- model_name: gpt-4o
  litellm_params:
    model: openai/gpt-4o

- model_name: claude-sonnet-4-6
  litellm_params:
    model: anthropic/claude-sonnet-4-6

This repo includes a Claude Code skill at .claude/commands/llm-ops/gateway/. Open the project in Claude Code and run /llm-ops:gateway to diagnose, install, or configure the stack conversationally. The bundled diagnostic checks Docker, containers, API health, credentials, and environment variables in one pass.

source scripts/gateway-env.sh

# Tools that use the OpenAI or Anthropic SDK now route through LiteLLM:
# ANTHROPIC_BASE_URL=http://localhost:4000
# OPENAI_BASE_URL=http://localhost:4000/v1

See examples/transparent-routing.md for permanent shell setup, per-tool virtual keys, and SDK examples. Tool-specific config examples are in examples/.claude/ and examples/.codex/.

flowchart LR
    Key["Virtual Key alias · tags · user_id"] --> Enrich["langfuse_enrich.py
Runs on every request"]
    Enrich --> Name["Trace Name
claude"]
    Enrich --> Session["Daily Session
claude-2026-03-14"]
    Enrich --> Tags["Tags
[claude]"]

Create virtual keys in LiteLLM Admin:

The enrichment hook automatically groups traces into daily sessions (e.g., claude-2026-03-14) — one per tool per day.

# Deletes: spend logs (30d), error logs (14d), traces (30d), scores (60d)
./scripts/prune-postgres.sh

# Requires: uv (https://docs.astral.sh/uv/)
uv run --script scripts/eval-session.py --today claude
uv run --script scripts/eval-session.py claude-2026-03-14
uv run --script scripts/eval-session.py claude-2026-03-14 --dry-run --verbose

./scripts/refresh-credentials.sh

./scripts/start.sh         # Start all services (with validation)
./scripts/stop.sh          # Stop all services
docker compose logs -f     # Follow all logs
docker compose ps          # Check service status

Every service is locked down by default:

• OrbStack (recommended) or any Docker-compatible runtime — OrbStack uses less memory and starts faster, which helps with an 8-service stack
• ~4 GB RAM available for containers
• Credentials for at least one LLM provider

.
├── .claude/commands/llm-ops/gateway/  # Claude Code skill (diagnose, install, configure)
│   ├── SKILL.md                       # Skill definition and instructions
│   ├── scripts/diagnose.sh            # Automated diagnostic cascade
│   └── references/topology.md         # Service topology and known issues
├── docker-compose.yaml                # 8-service stack definition
├── Caddyfile                          # Reverse proxy config (SSE, timeouts, logging)
├── .env.example                       # All required environment variables
├── litellm/
│   ├── config.example.yaml            # LiteLLM model config template
│   └── langfuse_enrich.py             # Trace enrichment hook (auto-loaded by LiteLLM)
├── examples/
│   ├── .claude/settings.json          # Claude Code proxy config
│   ├── .codex/config.yaml             # Codex CLI proxy config
│   └── transparent-routing.md         # Guide: route any tool through the gateway
└── scripts/
    ├── start.sh                       # Startup with validation
    ├── stop.sh                        # Graceful shutdown
    ├── gateway-env.sh                 # Source to route tools through the proxy
    ├── refresh-credentials.sh         # Check/refresh provider credentials
    ├── prune-postgres.sh              # Data retention maintenance
    ├── prune-postgres.sql             # PostgreSQL pruning policy
    └── eval-session.py                # LLM-as-judge session evaluation