feat: hosted-launch prerequisites — user-create, tenant-owned LLM keys, daily spend limit (#1303) - #1343
Conversation
Registration closes after the bootstrap user and no invite path existed, so a second account was impossible. The command is offline and direct-DB, the same trust boundary as set-password (#919): only someone with access to the server's database can run it. Enforces the #1285 password policy, refuses duplicates case-insensitively, and grants admin only with --admin.
Since #898 only superusers hold admin and since #1264 a hosted tenant reads only its own stored key, so admin-only key storage left a tenant with no way to enable LLM features. PUT/DELETE of LLM keys now need write scope and write the caller's per-user store; the GitHub PAT stays admin-only. A non-admin's manager skips the machine-wide migration: it copies the operator's keys into the caller's store, which would hand a tenant the operator's key. Two scope tests used the invalid path /keys/openai, which only 403'd because admin ran before provider validation; they now target GIT_GITHUB, the credential that is still admin-only. The web UI gates only the GitHub token slot.
CODEFRAME_USER_DAILY_COST_LIMIT_USD caps what each server principal spends per UTC day. Spend is today's token_usage across the workspaces the principal owns plus the one being run. Task start/resume, execute, approve+start and batch resume return 429 SPEND_LIMIT_EXCEEDED before writing any state; a batch rechecks before spawning each task. What is left clamps the workspace max_cost_usd, which is tenant-writable and so can never lift the limit: the single-task path passes it to execute_agent, a batch child receives it via CODEFRAME_RUN_COST_CEILING_USD. The auth-off operator is never limited.
- The daily remainder now limits NEW spend: it is offset by the task's prior spend instead of compared against it, so a resumed task is no longer refused at once (codex P2). - Runs hold a slice of the remainder while in flight and release it when they end, so concurrent starts and parallel batch siblings cannot each spend the whole of it (codex P1). A parallel batch splits it across its slots. - Every workspace a principal starts work in is recorded (workspace_spend_users) and counted, so moving to an unowned workspace no longer resets the meter (internal review). - The GitHub router built its credential manager before its admin check, so a refused non-admin connect still migrated the operator's keys into the tenant's store. Both routers now share one dependency in ui.dependencies. - cf auth user-create refuses a non-admin first account: the #898 backfill would promote it to admin and close web sign-up for the operator. Test doubles updated for moved names only: the offload fakes take spend_paths, the scoping tests patch CredentialManager at its source, and the plan-engine inspection test follows load_prior_task_cost into _load_prior_cost, which run and resume now call.
The workspace-switch test was refused by the first run's unreleased hold, not by the recorded spend, so it passed with recording disabled. The parallel share was only tested with the dict pre-filled, never through execute_batch.
…readable ledger Codex round 2. A batch reserved against the workspace list taken when it started, so spend in a workspace the user began using mid-batch was not counted; check_spend_limit now returns a spend_scope callable that the conductor calls before each task. And a spend DB that exists but cannot be read was counted as $0, handing out budget that may already be spent; it now refuses instead. A missing DB is still skipped (a deleted workspace must not block its user for good).
|
Cross-family review — codex (post-PR pass at 284b982)The new daily limit can be bypassed through delegated engines, and reservations do not respect the promised UTC-day reset. Both issues undermine the limit's execution behavior. Full review comments:
Earlier pre-PR rounds (codex r1: 2 findings, r2: 2 findings) and the internal review are triaged in the PR body; all were fixed. These two are being fixed now. |
…midnight Codex post-PR pass on #1343. - A delegated engine's spend never reaches token_usage and cannot be clamped, so under a limit it was unlimited. The REST routes already accept only react/plan; the conductor now refuses a delegated-engine task (e.g. a resumed CLI-created batch) while a limit applies. - Holds are tagged with their UTC day and only today's count, so a run that started before midnight no longer blocks the new day.
|
Triage of the codex post-PR pass:
Both are covered by tests that fail under mutation. |
|
|
Claude finished @frankbria's task in 12m 57s —— View job GLM precision review — PR #1343 @ eaa4913Bug-hunting pass over the full diff (logic errors, security, data loss, races — style/coverage excluded). Every suspicious hunk was verified against the surrounding code and its callers in the checked-out repo; earlier codex/internal findings and their fixes were re-checked and not repeated.
2 findings, both minor (posted as inline review comments):
Verified sound (checked, no defect): the 429 passes through every route's broad |
Demo (eaa4913): real
|
| Criterion | Action | Outcome evidence | Status |
|---|---|---|---|
| Admin-only user creation | cf auth user-create alice@… on a fresh DB |
exit 1: "No admin account exists yet, so the first account must be one: re-run with --admin" | VERIFIED |
… operator@… --admin, then … alice@… |
"Created operator@example.com (id 2, admin)", "Created alice@example.com (id 3, regular user)"; user-list shows the Admin column set for the operator only |
VERIFIED | |
alice logs in through /auth/jwt/login |
200; /auth/me returns is_admin=False |
VERIFIED | |
| Tenant owns their LLM key | alice runs PUT /keys/LLM_ANTHROPIC |
200 stored=True source=stored last_four=0000 |
VERIFIED |
| GitHub PAT stays admin-only | alice runs PUT /keys/GIT_GITHUB and POST /integrations/github/connect |
403 'admin' scope required; 403 |
VERIFIED |
| Operator key never copied to a tenant | the machine-wide store holds an operator LLM_OPENAI key; alice runs GET /keys after both writes |
LLM_OPENAI stored=False source=none |
VERIFIED |
| Daily ceiling enforced at task start | limit $1, $1.00 already spent; POST /tasks/{id}/start?execute=true |
429 SPEND_LIMIT_EXCEEDED "…$1.00 of $1.00 used today (UTC)…"; the task is still READY with 0 runs |
VERIFIED |
| Per user, not per workspace | the same user starts a task in a fresh workspace, other |
429 (spend in proj counts) |
VERIFIED |
| Clamps the workspace cap | $0.25 left, no max_cost_usd set: start → 200, run DONE. Clamp shown directly on the same task and LLM script, run with $0.05 left vs. no limit |
with $0.05: 1 LLM call, task BLOCKED with cost_cap_exceeded; without a limit: 2 calls, DONE |
VERIFIED |
| Hosted mode uses only the stored key | shipped in #1264 / PR #1327 (demo is on that PR) | n/a | VERIFIED (prior PR) |
Full transcript: tasks/demo-evidence-1303.txt (local; tasks/ is gitignored).
Review: #1343 (hosted-launch prerequisites)Went through the full diff (spend_limit.py, cost_tracker/agent/react_agent/runtime/conductor wiring, the shared credential-manager dependency, the settings/github scope changes, A few things worth a look: 1. Parallel/auto batch share is sized by configured
|
Review (feat/issue-1303)Scope is large but well-factored. I focused on the new Finding: inverted allowlist in
|
|
Review — PR #1343 (hosted-launch prerequisites) I read through Confirmed finding: parallel-batch share division compounds instead of splitting evenly
```python codeframe/core/spend_limit.pydef reserve_today_usd(user_id, repo_paths, share=1):
Total held is about $6.84, not $10, and the 4th sibling gets about 10.5% of the daily remainder instead of the intended 25% — more than 2x smaller than slot 1, purely as a function of thread-scheduling order inside This looks like it slipped through: Suggested fix direction: compute the wave's per-slot allocation once (e.g. snapshot Everything else checked out
Minor / low-severity, not blocking In Nice test discipline overall (mutation-tested, explicit triage tables in the PR body). The parallel-share issue above is the one thing I'd want addressed or consciously deferred before relying on |
Bot review on #1343 (GLM precision review + claude-review): - Parallel siblings each took 1/share of what the previous one left, so a 4-slot wave held 2.50/1.88/1.41/1.05 of 10. Siblings are now counted under the reservation lock and the remainder divided by the free slots; the slot count is min(max_parallel, task count). - Since #963 the auth-off principal carries the operator's real user_id, so the 'user_id is None' exemption never fired; type == 'disabled' is exempt. - The PAT gate was a denylist of GIT_GITHUB; it is now an allowlist of the two LLM providers, so a provider added later defaults to admin-only. - Recorded spend is read outside the hold lock (holds are what it guards). - test_concurrent_write_guards_1085 patched CredentialManager on the settings router, which no longer constructs it; it now patches the source. test_parallel_batch_child_gets_its_slots_share asserted the old 1/max_parallel split for a one-task batch; replaced by a two-task batch asserting an even split, the contract the review asked for.
|
Triage of the bot reviews at eaa4913 (GLM precision review + claude-review), fixed in b8a1f39:
CI's 3 failures ( |
|
Claude finished @frankbria's task in 11m 38s —— View job GLM precision review — PR #1343 @ b8a1f39Bug-hunting pass over the full diff (logic errors, security, data loss, races — style/coverage excluded). The newest commit (b8a1f39) reworks the parallel-share division, slot sizing, auth-off exemption, provider gate and lock scope, so those got the closest look; earlier-commit findings already posted by GLM/claude-review are re-checked only where the fix touched them, and not repeated.
1 finding, minor (posted as an inline review comment):
Verified sound (checked, no defect) — with the four b8a1f39 fixes each confirmed against real code:
Not repeated: the auto-strategy/ |
Review (post-b8a1f39, after the parallel-share / auth-off / PAT-allowlist / lock-scope fixes)This PR has already been through an unusually thorough gauntlet (codex x3, GLM, claude-review x3), and I re-verified the fixes from the latest round land correctly in the checked-out code rather than just re-reviewing the diff:
Two small residual items, both conservative (fail-closed) rather than exploitable, so not blocking:
Everything else I looked at (auth |
…release Bot review of b8a1f39 on #1343: - The sibling count ignored the UTC day, so a hold from yesterday took a slot while adding $0 to today: a wave across midnight over-committed (GLM). - Shares used the whole batch's task count, but dependency groups run in turn; each group now sets its own share, so a linear chain is not split max_parallel ways (claude-review). - release matched by amount alone and could drop another batch's hold of the same size; it now matches the group too (claude-review).
…replaced Mutation check showed it was dead: every parallel group now sets its own share before it runs, so the min(max_parallel, task count) sizing never took effect. A batch starts at one slot, which is right for serial execution.
The chain test only proved one-slot groups, which the default already gives, so dropping the per-group reset survived mutation.
|
Claude finished @frankbria's task in 11m 49s —— View job GLM precision review — PR #1343 @ e5694adBug-hunting pass over the full diff (logic errors, security, data loss, races — style/coverage excluded). The three newest commits (e85b6b8, 35a7e97, e5694ad) rework the per-group share sizing, the day-aware sibling count and the group-matched release, so those got the closest look; findings already posted by GLM/claude-review/codex for earlier commits were re-checked only where these commits touched them, and not repeated.
✅ GLM review: no defects found. Verified sound (checked against the real code, no defect) — the three newest fixes first:
Not repeated (already posted or documented in the PR body): the |
|
Triage of the reviews at b8a1f39, fixed in e85b6b8, 7e… (HEAD e5694ad):
Mutation check: 20/20 caught on the current code. The earlier |
Review (HEAD: e5694ad)This is the fifth pass I've made on this PR across its review rounds (codex x3, GLM x2, claude x4 including this one), and at this point I'm reviewing primarily to confirm the latest fixes land correctly rather than hunting for new ground. Both outstanding findings from the last round are now fixed and tested:
Also resolved since my last pass: the "share sized by configured Nothing new found. I re-checked the surrounding invariants this round touched (even-split math, lock scope, day-tagging) and they're unchanged and still sound per the prior rounds' sign-off. The two previously-noted, consciously-deferred items remain in that state and don't need re-litigating:
This PR is in good shape to merge. The review loop here (codex → internal → GLM → claude, repeated) has converged: each round's findings are narrower and lower-severity than the last, and this round's fixes come with tests that would fail under mutation if the fix regressed. |
|
Post-merge disposition: filed #1344 ([P2.71] server-side spend ledger: tenant-writable data, workspace attribution, per-process holds, delegated engines), #1345 ([P2.72] spend limit on PRD/discovery/task-generation/session chat), #1346 ([P3.39] key provenance in LLM 401 messages). Load-timing flake noted on #1326. Lessons: 4 entries appended to tasks/lessons.md. Dropped as trivia: the Thread.start() hold-leak window (claude-review, vanishingly unlikely) and a USER_CREATED audit row for |

Summary
Implements #1303: the hosted-launch prerequisites that were still missing. None of them blocks self-host.
cf auth user-create EMAIL [--name] [--admin].set-password([P1.1] Fix API-key and account lifecycle: keys survive deactivation, legacy bcrypt always fails, prefixes collide, no password reset #919), so only someone with access to the server's database can run it.--admin: the [P0.4] Enforce real scopes and tenancy: JWT sessions get admin, read-only keys can revoke keys, workspace ownership is reassignable #898 startup backfill would promote that account to admin and close web sign-up for the operator.PUT/DELETE /api/v2/settings/keys/LLM_*now need write scope and write only the caller's per-user store.connectstill copied the operator's keys. Both routers now use one shared dependency inui/dependencies.py.CODEFRAME_USER_DAILY_COST_LIMIT_USD(unset or ≤0 means off).SPEND_LIMIT_EXCEEDEDbefore any state is written. A batch checks again before starting each task.token_usageacross every workspace the principal has started work in (newworkspace_spend_userstable), plus every workspace it owns.max_cost_usd. The workspace value can be set by the tenant, so it can never raise the limit. The clamp limits new spend only: it is offset by the task's earlier spend, so a resumed task is not refused straight away.user_id is None).cf auth setupare never used by any LLM call #1264 / PR fix(llm): use stored API keys, not just the environment (#1264) #1327.Acceptance Criteria
cf auth user-create).max_cost_usd.cf auth setupare never used by any LLM call #1264).Test Plan
tests/core/test_spend_limit_1303.py(36),tests/ui/test_spend_limit_routes_1303.py(13),tests/ui/test_tenant_owned_keys_1303.py(8),tests/auth/test_user_create_1303.py(14), plus a newKeySlotjest case.Review triage
min(workspace cap, prior + remaining)in both enginesspend_scopecallable is re-read before each taskworkspace_spend_usersrecords every workspace the user starts work inuser-createwithout--adminon a fresh instance would be promoted by the backfill--adminuser-createwrites no audit rowset-password/activate/deactivate, which have no actor id offline. Left as isKnown Limitations / Intentionally Deferred
.codeframe/state.db, which agent code in that workspace can write. A tenant can therefore delete rows, or delete the DB, to lower its meter. A server-side spend ledger is the real fix and is filed as a follow-up. Hosted execution is still refused ([P1.47] Hosted mode is not a tenant boundary: terminal and agent processes run as the server user, contradicting SECURITY.md #1266) until per-tenant OS isolation exists ([P2.69] Agent subprocesses can still read same-uid secrets outside CodeFRAME's own processes: ancestor shell environ, codex workspace-write reads #1322).user_idcolumn ontoken_usagewould make it exact.max_cost_usdtoday. Single-task routes accept only react/plan.map_provider_errorkey provenance (from the issue thread, diagnostic text only). Follow-up issue.Implementation Notes
CREATE TABLE IF NOT EXISTS./keys/openai. They only got 403 because admin was checked before the provider was validated. They now targetGIT_GITHUB.CredentialManagerat its source, because the dependency moved.spend_scope.load_prior_task_costinto_load_prior_cost.Closes #1303