Adhar AI — The Agentic Layer of the Adhar Platform

Adhar • Built with ❤️ for developers!

Sanskrit: अधार (Adhāra) – Foundation

Adhar AI is the agent runtime behind the Adhar platform's ai/adhar-ai package (ADR-0024). Configure one LLM key and the platform gains the ability to investigate, explain, scaffold, provision and remediate — through its existing control surfaces, never around them.

Operating an internal developer platform demands expert fluency across Kubernetes, ArgoCD, Gitea, Crossplane, Keycloak, Vault, the LGTM stack, Kyverno and OpenCost. Adhar AI is how the platform answers for itself. Ask it why an application is degraded and it reads the real pods, the real logs, the real sync status and the real burn rate, then tells you — citing every source. Ask it to fix the problem and it opens a pull request.

⚠️ Active development. APIs and tool schemas may change. Images are published to GHCR as ghcr.io/adhar-io/adhar-ai-{runtime,mcp-<domain>} — see Project status.

Read tools read. Write tools open a pull request. Nothing applies to a cluster.

An agent holding kubectl apply and a cluster-admin token is a catastrophe waiting for a prompt injection. Adhar's whole thesis is GitOps-first, policy-enforced, RBAC-scoped, reviewable change, so an AI layer that bypassed that thesis would be a regression rather than a feature.

Every write tool here commits to a branch in Gitea and opens a PR against the GitOps repo. There is no kubectl apply path in this codebase, and the ServiceAccount the platform binds is read-only. A human merges, ArgoCD reconciles, and the change is as reviewable and revertible as any other. Every artifact the agent creates carries adhar.io/origin: adhar-ai and a branch prefixed adhar-ai/.

The property is structural, not configured. A test asserts, at the source level, that every write tool's body reaches open_pr and contains no apply/kubectl/patch_/delete_ call, and that the PR module imports no Kubernetes, AWS, Azure or GCP client at all. The agent's authority is exactly a contributor's, and it cannot be raised by a setting.

One image, three entrypoints:

In the platform, LLM traffic and the federated MCP endpoint are served by agentgateway rather than by the gateway in this repo (ADR-0025); the bundled gateway stays as the dependency-free local-dev fallback.

adhar-ai tools prints this as JSON; it is the same contract as contract/tools.json, and CI fails on any drift between the two.

35 tools across 8 domains, 4 of them writes. There is deliberately no kubectl_apply, argo_sync, helm_install or cloud-mutation tool, and a test asserts each of those names is absent from every domain.

delivery is the newest domain and the one that closed the widest gap. The others could say a Deployment was unhealthy, an Application OutOfSync, an image refused by policy — none could say why the build that should have produced that image failed, which is the question developers ask most. It reads Tekton runs, kpack builds, Kargo promotions and preview environments, and build_logs resolves a run name to the pod, then to the failing container, and tails it. Every package behind it is optional, so a platform without Tekton gets installed: false naming the package to enable — a fact the model can tell apart from "nothing is failing".

Outward, the eight servers are one federated endpoint — https://mcp.<host>/mcp on agentgateway, which multiplexes them all, validates the Keycloak token and authorizes per tool (read tools need platform-developer, the PR-opening tools need platform-admin). External agents — Claude Code, an IDE, ChatOps — drive Adhar through exactly these governed tools with one URL and one token. Tool names arrive prefixed by their server (gitops_propose_change), which is what the gateway's authorization rules key on.

The servers themselves no longer validate tokens: that control moved to the data plane, once, in front of all eight. They still receive the caller's bearer token (agentgateway sets preserveToken: true) so RBAC-scoped reads run as the user.

👉 Full reference: docs/TOOLS.md

The runtime reads its stage from the adhar-ai-config ConfigMap and never acts above it. Each rung differs from the one below in a way you can observe:

Authority only ever narrows as it flows. The ConfigMap sets a ceiling, a request may ask for less, an operator policy may be stricter still, and an unauthenticated caller is pinned to read-only regardless. No path widens it, and there is no rung at which the model chooses its own scope — the allow-list is read from the ConfigMap, never from a tool argument, because an argument is something a prompt injection can set.

Five operators run on events rather than a schedule: alert-triage (Alertmanager webhook), drift-explain (ArgoCD OutOfSync), cost-advisor (OpenCost), upgrade-preflight, and build-failure (Tekton CloudEvents) — the one a developer actually reads, because it arrives with the failing step already quoted. Each emits a structured finding. Whether that finding becomes a PR is the stage's decision, not the model's.

A single assistant with thirty-five tools answers every question the same average way. Eight specialists do not, and none of them can do more than the one assistant could — one per stage of platform work:

Routing is lexical and tuned on a real failure mode: guide used to list the natural ways to open a question ("what is", "where is", "how do i"), phrases scored double a word, and so the most common phrasings landed on the least capable agent — "what is k8sgpt reporting right now" went to the read-only teacher with two tools. Question form now scores a fraction of question subject, ties break on the more specific match rather than on roster order, and a question with no subject of its own still reaches guide.

Each agent's ceiling narrows and never widens: a read-only agent stays read-only for a platform administrator at scoped, because the narrowing belongs to the role rather than to the request. Routing is lexical, not a model call — spending a completion to decide who should spend a completion doubles latency on every request. An agent may hand work on, but only to a colleague it declared, only with a stated reason, and only four times before the chain is refused as a loop.

/chat and /tasks run the same roster. Both route, both apply the agent's ceiling, both scope the tools and the grounding. The only difference is the lifetime: /chat answers inside the request and writes no task row, because one row per chat message would bury the real work. POST /tasks returns an id immediately; the work runs behind a bounded worker pool, survives a restart where a database is configured, and reports its own state:

queued ──► planning ──► awaiting_approval ──► running ──► done | failed | cancelled

Above suggest, the agent writes a plan without acting and the task stops. Releasing it needs the same platform-admin credential the PR-opening tools do, because a task that can change the platform is exactly as privileged as a write.

Ten chores ship, all off and all in dry-run. Certificate expiry, drift, orphaned resources, failing scorecards, cost outliers, stale findings, cluster insights, failing builds, stale previews, runbook rot. Enabling one and letting it act are two separate decisions, and maxProposals bounds what a single run can open — the difference between a helpful Monday morning and an unreviewable flood.

The agent turns up where the work is: a Slack thread, a pull-request review that declares itself, a failed pipeline. A thread is a conversation, so a follow-up continues rather than starting fresh. Every inbound payload is treated as attacker-influenced, and the structural guarantee holds whatever it says.

It knows what it cannot answer. GET /capabilities is derived from the live tool surface rather than hand-written, so it never promises something that no longer exists. GET /coverage is the queue of questions the platform answered badly, clustered by how often each has been asked — which is the list somebody writes the missing runbook from.

👉 Full reference: docs/AGENTS.md

An assistant with thirty-five tools still fails the person who never finds out that any of it applies to them. Two failures look identical from outside: the capability is absent (nobody wired a tool, which is what every question about a failed build used to be), or it is unreachable — it exists and the person asks a colleague instead, while usage metrics say everything is fine.

GET /assist is the map that makes both visible: a persona × stage matrix where each cell names the agent, the tools, a question that really routes there, and the surfaces it works on.

curl -sS "$RUNTIME/assist?persona=developer&stage=build"
curl -sS "$RUNTIME/assist?entrypoint=chore"     # what arrives without being asked
curl -sS "$RUNTIME/capabilities"                # …plus `forYou`, for the caller's own token

Each persona is inferred from the caller's Keycloak groups, then refined by what they just asked: an admin asking a cost question is being a finops person for the next five minutes, and those are the offers worth showing them.

It is a gate, not a brochure. tests/test_assist.py asserts that every offer names an agent that exists and tools that exist, that it only names tools its own agent may call, that every example question actually routes to the agent it names (or to one that can hand off to it), and that gaps() is empty — every persona helped at every stage it claims. Add a persona, delete a tool or narrow a roster and the suite fails with the offer that became a lie. gaps ships in the response too, so a hole is visible to a reader and not only to CI.

The answer contract in the system prompt covers three things, each of which was a complaint first:

• Process before answering. Start wide (insights, sync_status, pipeline_runs — the thing you were asked about is often one of several with one cause), then narrow. Once there is a hypothesis, check the one thing that would disprove it. An empty tool result is not an answer: try the adjacent tool, then say what could not be established. "The build failed" is a restatement; "the go-build step exited 2 on a missing module" is an answer.
• Conversational, not a report. Plain prose, second person, no "Based on my analysis" — open with the finding. Bullets only for a genuine list; prose for anything with a because in it. One concrete offer of a next action is welcome; generic offers of further help are not.
• Accurate about what it knows. Observed, inferred and could-not-establish are three different statements, and the answer says which. Identifiers and error strings are quoted exactly, because a reworded error cannot be searched for. Every number carries its unit and window.

The scenario evals grade these as properties rather than phrasing — including, now, that an answer opens with its finding and contains real prose rather than only bullets.

The runtime is the one Adhar AI surface that does not sit behind agentgateway — it publishes its own hostname so the Console, the adhar ai CLI, Alertmanager and ArgoCD notifications can reach it directly. So it authenticates for itself:

With no credential at all, a caller is still answered — and pinned to read-only. Investigation stays open to anyone who can reach the port; authority requires a credential. Set ADHAR_AI_REQUIRE_AUTH=true to refuse the request outright instead, which is the right posture once the platform's own clients are wired up.

/healthz is the only one left open — it is the readiness probe, and it presents no credential. Everything else is gated, so ADHAR_AI_REQUIRE_AUTH=true closes every route that carries cluster detail.

👉 Threat model and controls: docs/SECURITY.md

An agent that works on a good day is a demo. These are the controls that make it safe to leave running on a bad one.

Two of these deserve a sentence each.

Duplicate delivery is real. Alertmanager retries, and so does ArgoCD notifications. Without deduplication one flapping alert becomes N agent runs and, above read-only, N near-identical pull requests for one problem. Operator events are keyed on the event body with volatile timestamps stripped, so a retry replays the first finding instead of doing the work again.

adhar_ai_denied_total is the security signal. A steady trickle of policy refusals is the system working. A spike is either a misconfiguration or something trying to make the agent exceed its authority — and it is the only metric that tells those apart from "the agent is quiet today".

👉 Full detail: docs/PRODUCTION.md

An agent that only knows Kubernetes in general is a search engine with extra steps. Adhar AI builds a knowledge base of your platform and keeps it current. Six sources, re-derived on a schedule:

Retrieval is hybrid. Vector similarity finds things phrased differently from the question; Postgres full-text finds the exact identifier the user typed (CreateContainerConfigError, CompositeCluster) that an embedding blurs away. Platform questions contain a lot of exact identifiers, so the two are fused by reciprocal rank rather than chosen between.

It learns. Notes are indexed the moment they are written, so a postmortem typed at 02:00 is retrievable at 02:01. Every operator finding is fed back, so the next similar alert retrieves what the last investigation concluded. And every answer returns the chunk ids behind it, so POST /feedback teaches the ranking what actually helped.

It is incremental, which is what makes keeping it current affordable. A refresh re-embeds only what changed — measured on Adhar's own corpus, a second pass over 1,299 chunks rewrote nothing and made zero embedding calls.

It degrades without a cliff:

So an unkeyed platform is still grounded. GET /healthz says which rung you are on, and every grounding block names its source and which path found it.

👉 Full detail: docs/KNOWLEDGE.md

local — no key. The model runs on the platform (ai/llm-d: vLLM behind the llm-d router with an agentgateway sidecar — Qwen3.6-27B-FP8 on one GPU, or Qwen2.5-0.5B on CPU) and agentgateway routes every local/* model name there. Both profiles answer to local/default. The hosted key stays primary; ADHAR_AI_LLM_SECONDARY_MODEL=local/default (set by the platform manifests) makes the self-hosted model the secondary every agentic feature falls back to when the primary fails — never on a budget or policy refusal — and /healthz → llm.answered_by says which one answered. ADHAR_AI_LLM_PROVIDER=local makes it primary instead.

Everything here speaks /v1/chat/completions and /v1/models to whatever gateway LLM_GATEWAY_URL points at, and always names a model in the body. Which backend that model reaches depends on which gateway is in front:

LLM_GATEWAY_URL may be given with or without the /v1 suffix — the platform sets it with, docker compose without, and the client normalizes it to exactly one /v1 either way. Token and request budgets are enforced in whichever gateway is in front, so a runaway agent costs a 429 rather than a bill.

uv sync --extra rag                    # Python 3.12+
uv run adhar-ai tools                  # the tool inventory — no cluster needed
uv run adhar-ai mcp --domain cluster   # :8081/mcp
uv run adhar-ai runtime                # :8082
docker compose up                      # the whole stack: gateway, all eight
                                       # MCP servers, the runtime and pgvector

Then ask it something:

curl -s localhost:8082/chat \
  -H 'content-type: application/json' \
  -d '{"prompt":"which applications are out of sync, and why?"}' | jq

👉 The full walkthrough — every entrypoint, a real cluster, an external agent, and what to expect at each step — is docs/GETTING_STARTED.md.

uv sync --extra rag
uv run pytest -q                       # 663 tests (686 with a database)
uv run ruff check src tests
uv run mypy src
uv run adhar-ai tools | diff -u contract/tools.json -   # the Go-CLI contract

The last one is not a formatting check. contract/tools.json is the schema snapshot the Adhar Go CLI is written against, so a diff there is a breaking change rather than a refactor.

Unit tests drive tools in-process and fake the toolbox, which is fast and blind to a whole class of defect. Every bug in the list below was found by something in this section while the unit suite was green, so both harnesses are worth running.

./hack/e2e-local.sh

Stands up all eight MCP servers and the runtime over real HTTP and asserts the seam between them: the transport's Host-header handling, the MCP client's result unwrapping, and whether an unconfigured backend's error text survives the trip to the model. A second runtime then runs against a scripted OpenAI-compatible server (hack/stub-llm.py), which drives the agentic layer end to end — a task that outlives its request, a handoff recorded on the task, a Slack thread that remembers its first message, a dry-run chore that proposes nothing. It needs no cluster, no key and no database: what it asserts is either true offline or honestly reported as unavailable, which is the property being tested. It runs in CI.

docker run -d --name adhar-rag -p 15432:5432 \
  -e POSTGRES_USER=adhar_ai -e POSTGRES_PASSWORD=adhar_ai \
  -e POSTGRES_DB=adhar_ai_rag pgvector/pgvector:pg16

ADHAR_AI_RAG_DSN=postgresql://adhar_ai:adhar_ai@127.0.0.1:15432/adhar_ai_rag \
  uv run hack/verify-knowledge.py --docs ../adhar/docs \
  --packages ../adhar/platform/stack/packages

Checks the knowledge base against a real pgvector: schema, ingestion from every source, incrementality, hybrid retrieval, exact-identifier lookup, immediate note availability, feedback ranking and deletion propagation. The same database makes the 13 skipped tests run:

ADHAR_AI_TEST_DSN=postgresql://adhar_ai:adhar_ai@127.0.0.1:15432/adhar_ai_rag \
  uv run pytest -q                     # 279 passed

LLM_GATEWAY_URL defaults to the platform data plane (http://adhar-ai-gateway.adhar-system.svc.cluster.local:8080/v1), which is unresolvable off-cluster on purpose — it is the production wiring, and every local entrypoint above overrides it. docker compose sets LLM_GATEWAY_URL=http://gateway:8080; for a bare uv run adhar-ai runtime, export the same.

Against a real platform, point it at the cluster's services:

export ADHAR_AI_ARGOCD_URL=http://argo-cd-argocd-server.adhar-system.svc.cluster.local
export ADHAR_AI_GITEA_API_URL=http://gitea-http.adhar-system.svc.cluster.local:3000/api/v1
export ADHAR_AI_PROMETHEUS_URL=http://prometheus-kube-prometheus-prometheus.adhar-system.svc.cluster.local:9090
export ADHAR_AI_LLM_PROVIDER=anthropic ANTHROPIC_API_KEY=...

Every setting has an ADHAR_AI_-prefixed name and an unprefixed fallback, which is how the platform manifests inject them. src/adhar_ai/config.py is the list, and docs/OPERATIONS.md is the annotated version.

./hack/release.sh                      # the next PATCH: X.Y.Z -> X.Y.(Z+1)
./hack/release.sh --dry-run            # the same checks, nothing written
./hack/release.sh 0.4.0                # a minor or major bump, named explicitly

One command publishes a version. It refuses unless you are on main with a clean tree in sync with origin; writes the new version into every place that states one (pyproject.toml, src/adhar_ai/__init__.py, the README badge, uv.lock); and then runs the same gates CI runs — ruff, mypy, the contract/tools.json snapshot, pytest. Only if all of them pass does it commit, tag and push. Any failure restores the tree, so a refused release leaves nothing behind — including the version bump it had already written.

A red ci on main is invisible until you try to release. images and ci are separate workflows, and images does not depend on ci: a commit whose lint or tests fail still builds, signs and publishes :latest, so the registry looks healthy while main is not. Worse, CI's steps are sequential — a lint failure skips type-check, tests, the contract snapshot and the end-to-end run, so one over-long line hides every other gate. That is how v0.3.3 was followed by three commits with a failing ci that nothing surfaced until hack/release.sh refused to cut 0.3.4. The habit that avoids it: run the gates above (or ./hack/release.sh --dry-run, which is the same set) before pushing, and treat ci on main as the release gate it is.

After the tag, images.yml builds and signs the nine images and release.yml then publishes the GitHub Release with the tool contract and the SBOM. The platform tracks :latest, so there are no manifest changes to make — roll the pods, and /healthz reports the version and commit now running:

kubectl rollout restart deployment -n adhar-system -l adhar.io/origin=adhar-ai

👉 Full detail: docs/RELEASING.md

The ai/adhar-ai package in the platform repo is opt-in and disabled by default, so the platform runs unaffected until someone supplies a key. When enabled it deploys the runtime and one MCP Deployment per domain (all this image, differing only by ADHAR_AI_MCP_DOMAIN), wires the adhar-ai-llm Secret through Vault and External Secrets, creates the Keycloak adhar-ai client, and applies a Kyverno adhar-ai-guardrails policy over what the agent may touch.

It does not deploy the gateway from this repo. The AI data plane is ai/agentgateway — upstream agentgateway, configured through Gateway API. It is the production data plane for both AI protocols:

Against the platform roadmap's Phase 3 agentic entry.

Exercised against a real LLM through the gateway, a real bootstrapping Adhar cluster and a real Gitea — not mocks:

• The agent diagnosed a live cluster. Three tool calls against a half-bootstrapped adhar-system, and every specific claim checked out: a missing argocd-redis Secret, Cilium returning putEndpointIdTooManyRequests, and a dex container exiting 20.
• It refused to invent data. Asked for CPU figures with no Prometheus configured, it named the backend and the environment variable that fixes it rather than producing a number.
• It opened a real pull request. Correct adhar-ai/ branch, adhar-ai label, provenance block with an audit id, commit trailer — and main untouched, which is the guarantee that matters.
• Embeddings are real. 1536 dimensions with genuine semantic structure: a paraphrase scored 0.39 against its original where unrelated text scored 0.12.

That run also found four defects the unit suite could not, all since fixed:

Still outstanding. The components have not yet run as Deployments in a cluster — the live run drove them from a workstation against that cluster's services. The images are published and the platform package already points at them, so this is a rollout rather than a build.

One caveat that applies to every Adhar-owned component, this one and the Console alike: the manifests track :latest with imagePullPolicy: Always, and an unchanged Deployment spec is not drift. ArgoCD will not restart pods when a new :latest is pushed, so picking up a new build needs kubectl rollout restart deployment -n adhar-system -l app.kubernetes.io/part-of=adhar-ai (or any change that recreates the pods). /healthz now reports the revision it is actually running, so the result can be checked rather than assumed.

Apache 2.0, same as the platform.