WARDEN

WARDEN is an AI on-call engineer for AWS. When an alarm fires, it gathers the evidence, finds the likely cause, writes a short report in Slack and proposes a fix. Nothing changes until that exact fix is approved: in production always by a person, with a passkey; in the dev environment by Eldane, a second AI that reviews the fix, unless a person says no first. Then WARDEN applies it, checks that it worked and undoes it if it did not, and every step is recorded.

1. An alarm fires. A CloudWatch alarm (or a Kubernetes alert) reaches WARDEN, and WARDEN opens an incident.
2. It collects the facts first. It reads the logs, metrics, recent deploys and recent changes, with read-only access, before the AI sees anything. Passwords, keys and personal details are blanked out.
3. The AI names a likely cause and one fix. The fix must come from a short, fixed list (restart, scale up, roll back a deploy, close stuck database sessions and a few more), and each claim must point at the evidence.
4. Ordinary code checks the answer. Fixed rules, not the AI, decide whether the fix may be offered, is too risky, or should go to a person with no fix at all.
5. It reports in Slack: what broke, the likely cause, the proposed fix and who should act.
6. The exact fix is approved. In production that is always a person, with a passkey on WARDEN's approval page (or a signing key on the command line). In dev, Eldane - a second Claude model that reviews the fix and is always labelled an AI - approves it alone when it agrees; the channel is told, and a person can deny it with their passkey for five minutes, and until the apply step starts. People can still discuss the incident in Slack and ask for a new plan. If anything the fix depends on changes first, WARDEN makes the plan again and asks again.
7. It applies the fix once, checks that the service recovered, and undoes the fix if it did not.

The whole system on one page: an alarm comes in, WARDEN's separate parts read the evidence, ask the AI, check its answer and report, and nothing changes until the exact fix is approved. Click a picture for a sharp copy.

One incident from the alarm to the signed record, step by step, with every safety check marked.

• Nothing is applied without a signed approval of that exact fix. The AI that diagnoses only suggests. In production the approval is always a person's. In dev it may be Eldane's, the AI critic's: its own zone, its own signing key, small reversible fixes only (tiers T1 and T2), never a change to WARDEN itself, never while the kill switch is on - and a person's "no" overrides it. A failed fix, a rollback, a person's veto, a change of either model or a missed catch trial (a planted bad plan Eldane said yes to) puts dev back to "a person approves" until a person signs it back on (warden autonomy enable, OPERATIONS.md).
• A kill switch. warden killswitch on stops every fix at once. Turning it off again needs a signed approval.
• Reading and writing are kept apart. The parts that gather evidence can only read. Writing uses a separate identity with tight limits: on AWS, a short-lived session for one approved fix; on Kubernetes, a deployment can be scaled up by at most two replicas.
• Text from the systems it reads is treated as data, never as orders. A log line or an alert can hide instructions planted by an attacker. WARDEN turns such text into numbered facts that the AI can quote but cannot take orders from; the code calls this quarantine. An optional local detector flags text that looks like an attack.
• Secrets are masked before any text leaves WARDEN, and checked again on the way out.
• Every step is recorded in a tamper-evident audit log. In the cloud runtime the log is signed with a key held by AWS KMS and anchored in write-once S3 storage.
• Spending limits. The AI's cost has a ceiling per incident and per day.

What this is, and is not. WARDEN is an engineering project that builds an AI incident responder with a person in control, end to end, and measures it honestly. Its ideas are not new: evidence first, fixed rules, human approval and least privilege are where the field is going (AWS DevOps Agent, HolmesGPT, Meta's DrP and incident.io's AI SRE work the same way). It is not offered as a product to adopt as-is. Read the benchmarks and the failure-mode register before trusting any part of it.

• No published run has applied a fix on a live AWS account. WARDEN has read a real AWS account, and its fixes are tested in CI against a real Kubernetes cluster (k3d) and real database servers.
• It can be wrong. The AI does not always give the same answer to the same evidence, and a confident, well-supported wrong answer can still reach the approver. That is why every production fix waits for a person.
• In dev, an AI approves another AI's fix. Eldane is the other Claude model (Opus 5.5 or Sonnet 5.5) from the one that diagnosed, on the same API key. Models of one family share blind spots and tend to prefer answers like their own; what is done about it - a blind first pass, citations checked in code, catch trials, demotion at the first failure, a person's veto - is in eldane.py. It has not yet approved a fix on a live account.
• It fixes only what is on its list. Anything else, including database failover, schema changes and anything that deletes, is handed to a person.
• It sees only what its access allows, and masking works from known patterns: a secret in a format no pattern knows is not caught.
• One person approves. This deployment's approver list has one person, whose approval decides alone; the code supports requiring two. dev, where the test bench runs, also has a second, labelled approver: a test key that Claude (the AI that helps build WARDEN) holds, scripts/bench_approve.py. It may approve only small, reversible fixes, only in dev (tiers T1 and T2, set by the owner on 2026-10-15), and every approval it gives is recorded as bench. WARDEN refuses the bench approver in any environment that is not of the nonprod tier, whatever its entry in the approver list says, and makes its key only for a nonprod environment.
• The measurements are limited: synthetic held-out incidents, single runs, and live reads from September 2026 under an older version. Details are in Status and limits.

No cloud account and no API key: mock mode runs the whole pipeline deterministically on five recorded incidents.

docker compose up                 # mock mode by default

or

pip install -e ".[dev]"
WARDEN_MOCK=1 warden demo --verbose
WARDEN_MOCK=1 warden run --incident inc-004 --report      # one incident, with its redacted report

warden run changes nothing: a live change goes only through the RemediationWorkflow.

With the Temporal CLI installed:

temporal server start-dev                                          # a local Temporal server
export WARDEN_TEMPORAL_KEY=$(python -c "import os,base64;print(base64.b64encode(os.urandom(32)).decode())")
WARDEN_MOCK=1 warden worker &                                      # --platform none: every live change is refused
printf '%s' '{"rule": "PodOOMKilled", "state": "ALARM", "transitioned_at": "2026-08-21T14:11:00Z", "alert": {"alert_id": "inc-002", "name": "PodOOMKilled", "severity": "high", "service": "checkout", "environment": "prod", "summary": "Repeated OOM kills", "started_at": "2026-08-21T14:11:00Z"}}' > event.json
WARDEN_MOCK=1 warden intake event.json                             # an alarm event: started, grouped or escalated

To let a worker change something, connect a platform (warden worker --platform k8s|db|aws|all) and approve a plan with warden status <workflow-id> and warden approve <workflow-id> --plan-hash <hash> --approver <id> --key ~/.warden/<id>.pem (key files are read only from the key folder, WARDEN_KEY_DIR, default ~/.warden).

docs/SETUP.md walks through the AWS roles, the GitHub environments, secrets and variables, Temporal Cloud and Slack. The runtime is a Terraform module, terraform/modules/warden-runtime; day-two operations (key rotation, the kill switch) are in docs/OPERATIONS.md, and what to do when WARDEN itself misbehaves is in docs/RUNBOOK-WARDEN-INCIDENT.md.

Ask a language model to fix an incident and it will always produce something. Given thin evidence it does not say "I don't know"; it produces a confident, plausible action. In a chat assistant that is a bug. In an operations tool it is an outage with a good explanation attached.

An agent that can act on production needs what operations software has always needed: evidence, proportionality, an audit trail, a cost ceiling and a person in the loop. WARDEN is an attempt to build those around one model call and to measure where they hold and where they do not.

alert -> gather evidence -> redact -> quarantine -> diagnose (one model call) -> verify -> escalate | plan a fix
                                                                                              |
                                        signed approval of the exact plan -> apply once -> check -> keep or roll back

1. Evidence first. Read-only backends gather logs, metrics, deploys and recent changes before the model runs, each under a hard timeout. The model does not choose what to look at.
2. Redaction and quarantine. Secrets and identifiers are masked before anything leaves the process, and raw log text is turned into numbered, typed facts. The model sees evidence as data, never as instructions.
3. One model call. The model returns a hypothesis with citations and one action from a closed list. It cannot invent delete_database.
4. A deterministic verifier. Plain Python rules (P0-P33) decide: reject, escalate to a person, or allow the proposal to go to an approver. Each verdict lists every rule that fired. New rules run in observe mode first (P25-P28 today): recorded on every verdict, obeyed by none, until they are measured.
5. A fix is a plan, not a command. An allowed proposal opens a plan from a fixed catalogue, made from what WARDEN reads live. An approver approves the plan's hash: in production a person; in dev Eldane, the AI critic, unless a person denies it first. WARDEN applies it once, checks that the service recovered, and rolls back if it did not.

Everything runs as Temporal workflows: IncidentWorkflow for a diagnosis and one RemediationWorkflow per target. The full design is in docs/ARCHITECTURE.md, and why the model never decides is argued in docs/ai-boundary.md.

Every box is a module or file in this repository, and every arrow is a call that exists in the code. Dotted arrows are optional (a flag, a setting, or a part hosted by the cloud runtime). tests/test_readme_architecture_r30.py checks that every module and activity named here exists.

flowchart TB
    subgraph IN[Ways in]
        ALARM[an alarm event<br/><i>warden intake</i>, <i>intake.py</i>]
        AM[Alertmanager webhook<br/><i>webhooks.py</i>: bearer token,<br/>size caps, still-firing check]
        MCP[warden-mcp<br/><i>mcp_server.py</i><br/>start_incident_diagnosis,<br/>request_remediation, workflow_status]
        CLI[warden run / demo<br/><i>cli.py</i>, in-process]
    end
    AM -.->|"host: API Gateway"| ALARM
    ALARM -->|"group repeats, flapping,<br/>open-incident cap"| IW

    subgraph TEMPORAL[Temporal - payloads encrypted by <i>codec.py</i>, API key + TLS <i>runtime.py</i>]
        IW[IncidentWorkflow<br/><i>workflows.py</i>]
        RW[RemediationWorkflow<br/><i>workflows.py</i><br/>one per target]
    end
    MCP --> IW & RW

    subgraph INC[Incident activities - <i>activities.py</i>]
        PREP[prepare: gather, redact, tripwire,<br/>evidence ids, quarantine to typed facts;<br/>reread: once more, when a source was empty right after the alert;<br/>investigate: read more of what the alert names;<br/>after_hold: does an alarm held as expected down still fire;<br/>plan_fix: an approved proposal opens its fix plan<br/><i>tools, redaction, tripwire, evidence, quarantine, investigation, resolver</i>]
        DIAG[diagnose: ONE model call<br/><i>llm.LLMClient</i> -> <i>providers.py</i><br/>Anthropic API in production; other providers selectable]
        VER[verify: rules P0-P33, P25-P28 observed only;<br/>second_opinion: a second diagnosis must agree<br/><i>verifier, grounding, numbers, language, decide</i>]
        NOTE[notify: report through the outbound gate;<br/>no_plan: says when no plan was opened, and why<br/><i>reporting, gate, chatops</i>]
        PREP --> DIAG --> VER --> NOTE
    end
    IW --> PREP
    CLI --> PREP

    subgraph SRC[Read-only evidence backends - <i>tools.resolve_backend</i>]
        FX[fixtures]
        AWSB[AWS: CloudWatch, ECS, Lambda, ...<br/><i>aws_backend, aws_stack</i>]
        K8B[Kubernetes<br/><i>k8s_backend</i>]
        DBB[five database engines<br/><i>database.py</i>]
    end
    PREP --> FX & AWSB & K8B & DBB

    subgraph REM[Remediation activities - <i>activities.py</i>]
        PLAN[resolve_plan: catalogue entry -> exact plan<br/>from live state, plan hash; environment<br/>and self-target checks <i>catalog, environments</i>]
        GATE[gate: bounds, cool-down, kill switch,<br/>change freeze, approval-fatigue cap<br/><i>bounds, freeze</i>]
        CRIT["critic_review, when the run has it (patched eldane-critic):<br/>Eldane, the AI critic, in its own zone, the other Claude<br/>model than the diagnosis, blind first pass; signs in dev at L2 (a person's<br/>no still wins) - else a person decides <i>eldane, autonomy</i><br/>critic_say / debate_reply: the debate in the discussion<br/>channel, labelled an AI, its own Slack token <i>eldane_slack</i><br/>autonomy_state / autonomy_enable: a person's signed<br/>re-enable after a demotion (AutonomyWorkflow)"]
        APPR[check_approval / check_passkey<br/>signed approval of the exact plan hash<br/>approval_wait: until when the run waits<br/>approval_extension: a wait an approver extended<br/><i>approvals, passkeys, extend</i>]
        APPLY[precheck - apply - check_success<br/>3 healthy checks, re-checks at T+15 and T+60<br/>record_result; rollback when not recovered<br/>side_effects: other alarms after the fix<br/>again_alert: diagnosed again after a relapse]
        FIN[finish: audit row, change record<br/>summary_owed: the incident's one message<br/>when a run ends before any said it<br/><i>changes.py</i>]
        PLAN --> GATE --> APPR --> APPLY --> FIN
        GATE -.->|"if the run has the critic step"| CRIT -.->|"Eldane's signed approval, L2 dev only"| APPR
    end
    RW --> PLAN
    APPR -.->|"announce: the request with the incident's summary,<br/>expired, how it ended; approved, denied and<br/>fixed kept on the records page (records.py)"| NOTE

    subgraph PEOPLE[People]
        SIGN[warden approve<br/>Ed25519 break-glass key]
        PAGE[approval page<br/><i>approval_page.py</i>: passkey,<br/>typed target]
        SLACK([Slack thread per incident,<br/>Teams, webhooks])
    end
    SIGN -->|signal approve| RW
    PAGE -.->|"signal approve_passkey<br/>host: the cloud runtime"| RW
    NOTE --> SLACK

    subgraph WRITE[Writes - only after a signed approval]
        KW[restart, bounded scale<br/><i>platforms/k8s.py</i>]
        AW[Lambda alias and concurrency, event source<br/>mapping, EventBridge rule, DynamoDB<br/>capacity, ECS rollback <i>platforms/aws.py</i><br/>per-plan actor session <i>identity.py</i>]
        DW[close idle-in-transaction sessions<br/><i>platforms/db.py</i>]
    end
    APPLY --> KW & DW & AW

    subgraph RECORD[The record]
        AUD[(audit: hash chain, signed checkpoints<br/><i>audit.py</i>)]
        ANCHOR[(S3 Object Lock anchors)]
        GH([GitHub issue per applied change])
    end
    INC & REM -->|every step| AUD
    AUD -.->|"KMS signer and anchor<br/>adapters, cloud runtime"| ANCHOR
    FIN -.->|"WARDEN_CHANGE_REPO"| GH

    subgraph DATA[Data, pinned by hash - <i>src/warden/data/</i>]
        D1[incident_signatures, environments,<br/>freeze, providers, decider]
    end
    D1 --> VER & GATE & PLAN

How a change reaches an environment: main runs the tests and deploys nothing; the dev and prod branches each deploy their own environment, without stored AWS keys, and infrastructure changes only from a reviewed plan.

Every component by its real name is in the detailed diagrams: the runtime and identities, environments and outside services. Their editable sources and how to re-export them are in docs/diagrams/.

Each control below is enforced in code and covered by tests; the limits are stated next to it.

Environments are data: src/warden/data/environments.yaml sets each environment's allowed actions and principals. An environment it does not list falls to a default that can only escalate.

Every fix waits for a signed approval of its exact plan. In prod that is always a person's. In dev it may be Eldane's, alone (autonomy: L2 in environments.yaml), or the labelled bench approver's, for T1 and T2; a person's "no" stops either until the apply step starts. Prod's ceiling is L1 - Eldane reviews and is scored, a person approves - and an L2 for a prod-tier environment is refused when the file is loaded.

Everything is set by environment variables. In the cloud runtime, plain values come from SSM Parameter Store and secrets from AWS Secrets Manager (src/warden/settings.py); each trust zone may read only its own secrets.

Models. WARDEN is not tied to one vendor: providers.py supports the Anthropic API, Amazon Bedrock, Gemini, OpenAI-compatible hosts (OpenAI, Groq, OpenRouter, Ollama for fully local use) and the claude command-line tool. But a model diagnoses only after it passes WARDEN's replay set: scripts/qualify_provider.py re-diagnoses 30 incidents recorded on a live AWS account, and only a model with no wrong answer let through and at least 17 correct is listed in src/warden/data/providers.yaml. Any other pair is refused. A row qualifies a model only on the prompt it was measured with: the pinned prompt is prompt.sha in the same file.

Qualified on the pinned prompt (as of 2026-10-21, from src/warden/data/providers.yaml): none. Every model is refused, and each incident gets the rules-only diagnosis and goes to a person.

The last model to pass, Claude Sonnet 5.5 (17 of 30 on 2026-10-15), passed on an earlier prompt (d435b59f), which was replaced on 2026-10-17, so it does not count. A model is qualified on the pinned prompt with one command, scripts/qualify_cloud.py (BENCHMARKS.md), which writes the result and this status line; until a model passes on it, production runs on the rules alone.

Keys and connection strings are read from the environment. Put them in a gitignored .env file or export them from a secret manager; never type one on the command line, where it stays in the shell's history.

Examples of each evidence source:

pip install -e ".[aws]"; WARDEN_BACKEND=aws WARDEN_AWS_CLUSTER=<cluster> warden run --incident inc-002 --service <service> --started-at now
pip install -e ".[k8s]"; WARDEN_BACKEND=k8s warden run --incident inc-002       # the cluster your kubeconfig points at
pip install -e ".[postgres]"; WARDEN_BACKEND=postgres warden run --incident inc-005   # WARDEN_DB_DSN exported

For Kubernetes, kubectl apply -k k8s/ installs a read-only ServiceAccount (get and list only, bound per namespace) and k8s/job.yaml runs one diagnosis as a Job under the restricted Pod Security standard. The separate write identity is k8s/remediation-rbac.yaml: get and patch deployments, held by an admission policy to the replica count and the restart annotation. In the cloud runtime the read and act zones connect Kubernetes and the database too (--platform aws,k8s,db): reads in each watched environment's platform-reader role (view-only on EKS; the database's pg_monitor login), every write in the approved plan's own actor session (on EKS the remediator group, bound in shop only; on the database the warden_<env>_terminate login, an IAM token and no password, its server's certificate verified, on the cluster the approved plan was read from). An environment with no EKS cluster (WARDEN_EKS_CLUSTERS) or database (WARDEN_DB_ENVIRONMENT) configured reads and changes nothing there; its plans go to a person.

pip install -e ".[mcp]" and warden-mcp expose the policy gate to any MCP client. Most MCP servers give an agent more capability; this one gives it a constraint. verify_remediation returns a binding verdict with the policy ids that fired, and every response carries may_execute: false. WARDEN_MCP_PROFILE sets what a client may do: read (the default) may start a diagnosis and read its workflows; remediate may also request a fix, which still needs a signed approval - in production a person's.

Version 0.10.0. Measured, as of 2026-10-16:

• Reading live systems. WARDEN has read a real AWS account: ECS with CloudWatch (42 runs), managed EKS (30) and RDS for PostgreSQL (18), all in September 2026, under an older version. Results: docs/BENCHMARKS.md.
• Generalising. On 172 synthetic held-out incidents written by independent authors, WARDEN chose the right action in 132-138 of 172 across three runs, against a floor of 103 for an answer that always escalates; the gate stopped all but one harmful or wrong proposal in each run. The cases are synthetic and the runs are single runs. Today's baseline, to beat after re-qualification: always escalating gets 103 of 172 right; WARDEN got 132-138 of 172 right actions (Claude Sonnet 5, 2026-10-10), and Claude Sonnet 5.5's answers, re-scored under the gate of 2026-10-15, let 0 harmful answers through. Both were measured on earlier prompts, not the pinned one, and must be measured again once a model is qualified on it.

• The AI critic. How often Eldane approves a right fix, catches a wrong or harmful one, and blocks a right one is not yet measured: with the cloud runtime's key, run python scripts/eldane_bench.py --model claude-opus-5-5 --cap-usd <cap> --out <a new directory>, after the same command with --dry-run in place of the last two (docs/BENCHMARKS.md).

• Choosing the model. A model diagnoses only once it is qualified on the pinned prompt (the status line under Models above); until then production falls back to the rules alone. Claude Sonnet 5.5 scored 17 of 30 on the replay set with no wrong answer let through on 2026-10-15, on an earlier prompt. The bar is 17.
• Applying fixes. The write platforms are exercised in CI against a real k3d Kubernetes cluster and real PostgreSQL, MySQL and SQL Server servers. No published run has applied a fix on a live AWS account.

Not done, or not measured:

• GKE and AKS have never been run; k3d runs in CI only.
• The in-cluster Job and the Kubernetes write path have not run inside EKS.
• A rollout undo on Kubernetes, clear_cache, database failover, schema changes and anything that deletes are not offered; failover is rejected by policy in production.
• MySQL, Redis, MongoDB and SQL Server are tested against CI containers, not managed services. Oracle is not supported.
• The eval suite tests deterministic behaviour, not live model quality.
• A timed-out reader thread is abandoned, not killed (Python cannot stop a thread); set a socket timeout on the backend client too.
• The model is not deterministic: the same evidence can get different answers on repeated runs.

The defects found so far, and what holds each, are in docs/FAILURE-MODES.md. The design choices and their costs are in docs/DESIGN-DECISIONS.md.

make check     # ruff, unit tests and the eval gate - what CI runs

Over 5,000 tests (tests/, kept a lower bound by tests/test_docs_honesty.py) and 26 evals (evals/: routing, safety invariants, cost, budget, audit trail), the poisoned-log eval (11 planted-log attacks a model follows, each run in dev and prod and followed two ways: 44 runs; results), plus 42 opt-in live-infrastructure tests (29 against a live Kubernetes cluster, 13 against five real database engines), which CI runs against k3d and service containers:

WARDEN_K8S_INTEGRATION=1 pytest tests/integration/test_live_cluster.py   # needs a cluster
WARDEN_DB_INTEGRATION=1 WARDEN_TEST_PG_DSN=postgresql://... pytest tests/integration/test_live_database.py

CI asserts the demo's actual verdicts, not just its exit code, and an integration step fails if anything in it skips. scripts/mutation_check.py breaks the safety code on purpose and requires the suite to notice.

Helios is a Rust and Z3 infrastructure failure simulator built on the same principle: a rigorous core around an AI shell. WARDEN's infrastructure deploy runs Helios on every Terraform plan and refuses a plan that fails under a zone outage or resource loss the stack survives today (scripts/helios_gate.py).

Report a vulnerability privately as described in SECURITY.md. WARDEN is released under the Apache License 2.0.

The optional injection detector's model is not part of WARDEN and is never bundled with it. Each deployment downloads Meta's Llama Prompt Guard 2 from Hugging Face under its own account, after accepting Meta's Llama 4 Community License itself.