docs(agents): lifecycle, report api, waits, context cli

- readme, changelog, claude.md, agent-automation.md, report schema - update task.md
codejunction · Oct 4, 2026 · Oct 3, 2026 · Oct 3, 2026 · Oct 4, 2026 · Oct 4, 2026
commit d63b246044c424caa3b42aeea018b884e459ea36
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,16 @@
 
 All notable changes to shelldeck are listed here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions follow [Semantic Versioning](https://semver.org/).
 
+## [Unreleased]
+
+### Added
+
+- Server-owned agent lifecycle: `idle`, `working`, `blocked` (with a reason), `done` and `exited`. An integration report outranks screen/activity detection, and when it expires the server falls back to detection again. The AI agents page shows each agent's state and its source. Needs-you push alerts fire only when an agent becomes blocked.
+- `POST /api/agent-reports`: integration reports tied to one terminal by its `SHELLDECK_AGENT_REPORT_TOKEN`, with validated state, TTL, display metadata and resume argv. Validated native agent sessions are stored in the new `agent_sessions` table.
+- `sd agent status`, `sd agent wait` (event-driven; a replacement agent process never satisfies an old wait) and `sd agent report`.
+- Persistent agent context (`context.db`): per-project task, state, memory, decisions, events and relationships, with FTS5 search across projects, secret redaction, stale-source detection, a context budget and Markdown projections in `.shelldeck/`. Commands: `sd context`, `resume`, `recall`, `memory`, `remember`, `discover`, `decide`, `decisions`, `task update`, `knowledge verify`, `projects`, `project` and `relate`.
+- Hand-off files include a context snapshot, and the installed agent skill teaches the context workflow.
+
 ## [0.0.6] - 2026-10-01
 
 ### Added
@@ -105,6 +115,7 @@ First public release.
 - Mandatory password with per-browser logins, idle lock, and HTTPS or SSH-tunnel remote access.
 - `sd` CLI with a startup banner, `sd search` (LLM-free code search), and one-line installers for Windows and Linux.
 
+[Unreleased]: https://github.com/codejunction/shelldeck/compare/v0.0.6...HEAD
 [0.0.6]: https://github.com/codejunction/shelldeck/releases/tag/v0.0.6
 [0.0.5]: https://github.com/codejunction/shelldeck/releases/tag/v0.0.5
 [0.0.4]: https://github.com/codejunction/shelldeck/releases/tag/v0.0.4

diff --git a/CLAUDE.md b/CLAUDE.md
@@ -89,6 +89,8 @@ uv tool install --force -e .        # global sd/shelldeck from this checkout (th
   - `screen` asks open browsers for a fresh xterm snapshot (`{"type":"snapshot"}` to the socket) and waits up to 1s, because raw TUI output is cursor-addressed. With no browser open it falls back to raw output.
   - `input` writes the text, waits 0.3s, then sends CR: TUIs take text plus CR in one burst as a paste.
 - **Terminal nicks.** `sessions.nick` is a person's name from `db.NICKS`, unique across all terminals (`_free_nick`, backfilled in `init_db`); `_session()` in cli.py resolves id, nick, name, then id prefix. PTYs get `SHELLDECK_NICK` and, for sub-agents, `SHELLDECK_PARENT` (`_attach` passes `extra_env`). `run()` exports `SHELLDECK_PORT` (and an absolute `SHELLDECK_HOME` when set), so `sd` in a terminal reaches the server that owns it.
+- `shelldeck/agent_state.py`: canonical lifecycle (`unknown|idle|working|blocked|done|exited`, `blocked_reason`), `Report`/`Registry` with source precedence (integration > custom > native > manifest > heuristic) and TTL expiry, and `parse_report()` validation (stable `ReportError` codes; resume argv must start with the agent's bare executable name and is accepted only from `integration:*`). The server owns state: `_check_agents` puts the old heuristic in as a `heuristic:*` report (`lifecycle.heuristic`, approval -> blocked/approval), resolves, and `_publish` broadcasts `agent_state` with `status` (legacy `state` kept) and pushes only on a transition into blocked. `POST /api/agent-reports` identifies the terminal only by `X-Shelldeck-Report-Token` (`report_tokens`, set as `SHELLDECK_AGENT_REPORT_TOKEN` in `_attach` and dropped when the PTY exits). `agent_kind` holds `(agent, generation)`; `/api/agent-wait` pins the generation and wakes on `_state_changed` events, never polls screens. The browser trusts server status only for authoritative sources (`AUTHORITATIVE` in app.js); otherwise its own screen heuristic stays.
+- `shelldeck/context.py`: persistent agent context in `<config>/context.db` (separate from shelldeck.db; FTS5 `search` table with a LIKE fallback). Projects are keyed by resolved folder (the deepest shelldeck project containing cwd, else the git top level). Everything stored goes through `redact()`; `SENSITIVE_FILE` paths are never referenced. Knowledge is de-duplicated per project by `_norm` and never merged across projects. `check_stale` re-hashes `source_references`. `project_context` returns a `ContextPackage` dict plus budgeted `text`. `project_files` writes STATE/TASK/MEMORY/DECISIONS.md (server runs it on a thread via `_project_later`). Server endpoints are `/api/context*`, wrapped by `_ctx` (sqlite errors -> 503 `context_unavailable`, never a crash). `_handoff_files` appends `context.snapshot`.
 - `shelldeck/team.py`: agent teams.
   - `tier()`/`pick_model()`: `--model auto|small|medium|large` maps onto the agent's own `models()` by regex (medium = its default).
   - `SPAWN` (agent -> model flag, args before the prompt) and `spawn_line()`/`kickoff()`: the kickoff prompt is limited to `SAFE_PROMPT` characters so one double-quoted string works in pwsh, cmd, bash, zsh and fish; the task itself lives in a file.

diff --git a/README.md b/README.md
@@ -97,6 +97,8 @@ sd ~/code/api      # add a folder as a project and open a terminal in it
 - **Agents that talk to each other.** From inside a terminal, `sd agents` lists the other agents in the project, `sd peek` reads another terminal's screen and `sd tell` types a message into it, tagged with the sender so it can reply. *Copy team prompt* on the AI agents page gives you text to paste into each agent so it knows how.
 - **Sub-agents and hand-offs.** `sd spawn "task" --model small|medium|large` opens a new terminal in the project with a sub-agent (Claude Code, Codex, Devin, Gemini, Qwen or opencode) already working on the task, picking a cheaper or stronger model by how hard the task is (`auto` guesses). It opens next to yours without taking the keyboard. `sd handoff Maya "task"` gives a task to an agent that is already running. The receiver runs `sd done <id> "summary"`, and the sender gets the summary typed in (when an agent runs there) plus a toast and chime. Every hand-off is written to the project's `.shelldeck/handoff.md` (git-ignored) and listed on the AI agents page. Sub-agents can't spawn more agents, and a terminal that closes mid-task marks its hand-off as exited.
 - **The parent is the control center.** A sub-agent never alerts you. When one stops on a question or permission prompt, shelldeck types the question into its parent agent's terminal; the parent decides and answers with `sd answer Maya 1` (keys for the menu) or `sd tell`. Only when the parent is a plain shell does the question come to you. Once a sub-agent's work is done, the parent asks you and closes it with `sd close Maya`, or use *Close* on the hand-off toast.
+- **One status, owned by the server.** Each agent terminal is `idle`, `working`, `blocked` (with a reason such as approval or question), `done` or `exited`. Every tab and the CLI see the same value, and the AI agents page shows where it came from: an integration report or screen/activity detection. `sd agent status` prints it. `sd agent wait Maya --until idle --timeout 10m` blocks until the agent gets there; it exits 2 on timeout and 3 if the agent exits or a different agent takes the terminal. Integrations report state with `sd agent report` (see [docs/agent-automation.md](docs/agent-automation.md)).
+- **Context that outlives the agent.** Each project keeps a task, its current state and next action, memory (facts and discoveries), decisions and recent events in a local database (`context.db` next to the shelldeck config). The same context is written as Markdown to `.shelldeck/STATE.md`, `TASK.md`, `MEMORY.md` and `DECISIONS.md`. A new agent runs `sd context` and continues where the last one stopped. `sd recall "auth architecture"` searches knowledge from every project, with its source project and files. Secrets are redacted before anything is stored, `.env`/key files are never referenced, and knowledge whose source files changed is marked stale. Hand-off files include a context snapshot. All of it stays on this machine.
 - **Agents know the commands.** On start, shelldeck installs a `shelldeck` skill for every agent CLI on PATH: a skill for Claude Code, Codex and Devin, and a marked block in the global instructions file of Gemini, opencode, Qwen, Amp, Droid, Copilot, Crush, Goose and Kiro. `sd install-skill --remove` takes it out and stops the reinstall; the AI agents page has an *Install skill* button per agent.
 
 <img src="https://raw.githubusercontent.com/codejunction/shelldeck/main/docs/assets/agents.png" alt="AI agents page: four named agents with their tool, model and project, two finished hand-offs with their results, and the available agent CLIs" width="100%">
@@ -185,6 +187,18 @@ sd done ID ["SUMMARY"] [--failed]    close a hand-off you were given; the sender
 sd answer TERMINAL KEY...            press keys in another terminal, e.g. answer a sub-agent's prompt: 1, y enter, esc
 sd close TERMINAL [--force]          close a terminal; from an agent only its own sub-agents (after you agree)
 sd handoffs [--all]                  hand-offs in this project and their status
+sd agent status [TERMINAL] [--json]  lifecycle state (idle/working/blocked/done/exited) and its source
+sd agent wait TERMINAL --until STATE [--timeout 10m]   block until an agent reaches a state
+sd agent report STATE --source S --agent A   report state from an integration inside a terminal
+sd context [PROJECT] [-q QUERY]      task, state, next action, memory, decisions, hand-off, git, events
+sd resume                            this project's unfinished task and where to pick it up
+sd recall QUERY                      search knowledge and decisions across every project
+sd memory [search QUERY]             this project's memory and decisions
+sd remember FACT / sd discover FINDING [--type T] [--file F]   add to project memory
+sd decide TITLE [-r REASON]          record a settled decision (sd decisions lists them)
+sd task update [STATUS] [--task --step --next --tests --error]   update the active task and state
+sd knowledge verify ID [--status S]  mark knowledge VERIFIED, REVIEWED, STALE or INVALIDATED
+sd projects / sd project NAME / sd relate PROJECT   known projects, one project, link related projects
 sd install-skill [AGENT...] [--remove]   teach agent CLIs the sd team commands (automatic on server start)
 sd edit FILE / sd view FILE          open a file in shelldeck's editor / viewer
 sd search QUERY [--root DIR]         LLM-free code search with ranked, highlighted snippets
@@ -201,7 +215,7 @@ sd reset-password                    emergency: forget the password (host only)
 - **Startup banner.** `sd` prints the version and the Local and Network URLs. When you start it from Win+R, the Start menu or a shortcut, its window stays open until you press Enter.
 - **Port.** The default port is `5455`. Change it with `--port` or `SHELLDECK_PORT`.
 - **Browser.** `--app` opens a chromeless Edge/Chrome window instead of a browser tab.
-- **Inside terminals.** Inside a shelldeck terminal, `SHELLDECK_SESSION_ID`, `SHELLDECK_NICK` (its name) and `SHELLDECK_PORT` are set, and `SHELLDECK_PARENT` in a sub-agent's terminal.
+- **Inside terminals.** Inside a shelldeck terminal, `SHELLDECK_SESSION_ID`, `SHELLDECK_NICK` (its name) and `SHELLDECK_PORT` are set, plus `SHELLDECK_AGENT_REPORT_TOKEN` (a per-terminal secret for integration reports), and `SHELLDECK_PARENT` in a sub-agent's terminal.
 - **Agent permissions.** Claude Code asks before running `sd done` and friends. To let sub-agents report back on their own, allow `Bash(sd:*)` in your Claude Code permissions.
 - **Code search.** `sd search` builds a persistent index and returns ranked snippets with confidence scores. Useful options: `--ext py,ts`, `--glob "src/*"`, `--top N`, `--format text|json|paths` and `--reindex`.
 

diff --git a/docs/agent-automation.md b/docs/agent-automation.md
@@ -0,0 +1,74 @@
+# Agent automation
+
+Shelldeck owns each agent terminal's lifecycle state. Scripts, parent agents and integrations read and wait on that state
+through the local authenticated API (the CLI sends the host token automatically).
+
+## States
+
+| State | Meaning |
+| --- | --- |
+| `unknown` | An agent is detected, but no source has reported yet. |
+| `idle` | Ready for a prompt. |
+| `working` | Running, reasoning or calling tools. |
+| `blocked` | Needs input. `reason` is one of `approval`, `question`, `authentication`, `tool_error`, `external_wait`, `unknown`. |
+| `done` | The task finished. An open hand-off still needs `sd done`; a report never closes one. |
+| `exited` | The agent process left the terminal. |
+
+Each status names its `source`:
+
+- `integration`: a built-in hook or plugin.
+- `custom`: a third-party reporter.
+- `native`: a direct agent API.
+- `manifest`: screen-detection rules.
+- `heuristic`: output activity and the question regex.
+
+The highest unexpired source wins. When it expires, the next one takes over.
+
+## CLI
+
+```sh
+sd agent status [TERMINAL] [--json]
+sd agent wait TERMINAL --until idle [--until blocked] [--timeout 10m] [--json]   # exit 0 reached, 2 timeout, 3 exited/replaced
+sd agent report working --source custom:mytool --agent codex [--reason R] [--title T] [--label L] [--ttl-ms 30000]
+```
+
+## API
+
+- `GET /api/agent-status[?target=<sid>]` returns `{agents: [{session_id, agent, generation, state, source, reason, detail, meta?}]}`.
+- `POST /api/agent-wait {session_id, until: [states], timeout: seconds}` returns `{result: reached|timeout|exited|replaced, status}`.
+  - The wait is pinned to the agent process present when it starts. A different agent later in the same terminal returns `replaced`.
+  - It wakes on state changes; it doesn't poll screens.
+- `POST /api/agent-reports` takes a report body (schema: [agent-report.schema.json](agent-report.schema.json)). It needs the
+  `X-Shelldeck-Report-Token` header with the terminal's `SHELLDECK_AGENT_REPORT_TOKEN`, plus normal API auth.
+  - The terminal comes from the token. A `session_id` in the body must match it.
+  - A report holds for `ttl_ms` (default 30 s, max 120 s), so reporters must refresh it.
+- Alarm socket messages: `{"type":"agent_state","session_id":...,"state":<legacy working|approval|idle>,"status":{...}}`.
+
+### Report errors
+
+All errors are `{"error": code}`.
+
+| Code | Status | Meaning |
+| --- | --- | --- |
+| `invalid_report_token` | 403 | The token is missing, or its terminal is gone. |
+| `session_mismatch` | 403 | The body names another terminal. |
+| `no_agent_running` | 409 | No agent process runs in the terminal. |
+| `agent_mismatch` | 422 | `agent` differs from the detected agent. |
+| `invalid_source` | 422 | Not `integration:`/`custom:`/`native:`/`manifest:` + id. |
+| `invalid_state`, `invalid_blocked_reason`, `invalid_ttl` | 422 | Field values are invalid. |
+| `invalid_metadata`, `invalid_tokens`, `invalid_detail` | 422 | Metadata is malformed (text is capped at 120 chars, at most 8 tokens). |
+| `invalid_agent_session_id`, `invalid_resume_argv`, `resume_agent_mismatch`, `resume_not_allowed` | 422 | Native session data was rejected. |
+
+Metadata (title, display agent, state label, tokens) is presentation only. It never changes state, notifications,
+hand-offs or restore.
+
+Native sessions (`agent_session_id` + `resume_argv`) are accepted only from `integration:*` sources and stored in
+`agent_sessions`. Automatic resume is not implemented yet; argv is never rebuilt from terminal output.
+
+## Context commands
+
+See the README's CLI list: `sd context`, `recall`, `memory`, `remember`, `discover`, `decide`, `task update`,
+`knowledge verify`, `projects` and `relate`. The matching endpoints are `GET /api/context`, `GET /api/context/recall`,
+`GET|POST /api/context/memory`, `POST /api/context/decisions`, `POST /api/context/state`,
+`POST /api/context/knowledge/{id}/status`, `GET /api/context/projects` and `POST /api/context/relationships`. Each
+takes `cwd`, `session_id` or `project` to choose the project.
diff --git a/docs/agent-report.schema.json b/docs/agent-report.schema.json
@@ -0,0 +1,30 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://github.com/codejunction/shelldeck/docs/agent-report.schema.json",
+  "title": "Shelldeck agent report v1",
+  "type": "object",
+  "required": ["source", "agent", "state"],
+  "properties": {
+    "session_id": {"type": "string", "description": "Optional; must equal the terminal bound to the report token."},
+    "source": {"type": "string", "pattern": "^(integration|custom|native|manifest):[a-z0-9_.-]{1,40}$"},
+    "agent": {"type": "string", "description": "Must match the agent process detected in the terminal."},
+    "state": {"enum": ["unknown", "idle", "working", "blocked", "done", "exited"]},
+    "blocked_reason": {"enum": [null, "approval", "question", "authentication", "tool_error", "external_wait", "unknown"]},
+    "detail": {"type": ["string", "null"], "maxLength": 120},
+    "agent_session_id": {"type": "string", "minLength": 1, "maxLength": 200},
+    "resume_argv": {"type": "array", "minItems": 1, "maxItems": 16, "items": {"type": "string"},
+                    "description": "argv[0] is the agent's bare executable name; no paths or control characters; 1024 bytes total."},
+    "metadata": {
+      "type": ["object", "null"],
+      "properties": {
+        "title": {"type": "string", "maxLength": 120},
+        "display_agent": {"type": "string", "maxLength": 120},
+        "state_label": {"type": "string", "maxLength": 120},
+        "tokens": {"type": "object", "maxProperties": 8, "propertyNames": {"pattern": "^[a-z][a-z0-9_.-]{0,31}$"},
+                   "additionalProperties": {"type": ["string", "number"]}}
+      }
+    },
+    "ttl_ms": {"type": "integer", "exclusiveMinimum": 0, "description": "Capped at 120000; default 30000."}
+  },
+  "dependentRequired": {"resume_argv": ["agent_session_id"]}
+}
diff --git a/docs/task.md b/docs/task.md
@@ -36,6 +36,9 @@
   startup/during-work/pre-stop context workflow (`sd context`, `recall`,
   `remember`, `discover`, `decide`, `task update`, `agent status/wait`)
   without exposing the database.
+- **2026-10-04 — Docs:** README, CHANGELOG (Unreleased), CLAUDE.md,
+  `docs/agent-automation.md` and `docs/agent-report.schema.json` describe the
+  lifecycle, report API, waits and context commands.
 
 The authoritative design is [AGENT_ENHANCEMENT.md](AGENT_ENHANCEMENT.md). This
 file is the execution checklist for breaking that design into reviewable,
@@ -414,7 +417,7 @@ temporary logging or exposing private terminal content.
 - [ ] Verify UI state text/icons with keyboard navigation and a screen reader.
 - [ ] Update `README.md`, `CLAUDE.md`, `CHANGELOG.md`, and the dedicated agent
   integration/automation/detection docs described in the design spec.
-- [ ] Document knowledge system, cross-project retrieval, and context CLI.
+- [x] Document knowledge system, cross-project retrieval, and context CLI.
 - [ ] Add release notes that list exact supported integrations and clearly state
   whether each supports detection, authoritative lifecycle state, and native
   restore.