Skip to content

feat(subagent): cooperative cancel checkpoint — make cancel actually stop thread-mode subagents - #3261

Merged
TimeToBuildBob merged 5 commits into
masterfrom
fix-3258-cancel-checkpoint
Jul 15, 2026
Merged

TimeToBuildBob merged 5 commits into
masterfrom
fix-3258-cancel-checkpoint

Conversation

@TimeToBuildBob

Copy link
Copy Markdown
Member

Problem

subagent_cancel() sent a SIGTERM / set a cancelled flag, but thread-mode subagents had no mechanism to observe it between LLM steps. The running thread would finish its current generation and potentially several more tool calls before the next natural completion point.

Solution

Adds a STEP_PRE cooperative checkpoint that fires before every LLM generation step in subagent threads. When a cancel is requested, it writes a {"op": "cancel"} record to logdir/control.jsonl; the checkpoint reads this file and if found: yields a partial-work note, writes cancelled status (first-writer-wins, so a natural completion can still win the race), then raises SessionCompleteException to exit chat() cleanly and release the semaphore slot.

Changes

gptme/tools/subagent/hooks.py — new _subagent_cancel_checkpoint()

  • Fires at STEP_PRE (before each LLM generation)
  • Two O(1) fast-path guards keep overhead minimal: thread-local agent_id is None check (no-op in parent), and a single stat() on control.jsonl (no-op when absent)
  • On cancel op: appends status message, sets cancelled result via set_subagent_result_if_absent (first-writer-wins), raises SessionCompleteException

gptme/tools/subagent/api.py — new _write_cancel_op() + updated subagent_cancel()

  • _write_cancel_op(): flock-safe JSONL append to logdir/control.jsonl (Windows-safe fcntl fallback, same pattern as prompt_queue.py)
  • subagent_cancel() in thread mode: calls _write_cancel_op then the existing cache-mark
  • subagent_cancel() in subprocess mode: calls _write_cancel_op before SIGTERM (so it works even if the subprocess ignores the signal)
  • ACP mode: unchanged (cache-mark-only, as before)

gptme/tools/subagent/__init__.py — registers cancel_checkpoint hook at STEP_PRE priority 100

tests/test_subagent_unit.py — 10 new tests

  • TestWriteCancelOp (3): JSONL written, appends correctly, creates missing logdir
  • TestSubagentCancelThreadWritesControlFile (2): thread and subprocess cancel modes write control.jsonl
  • TestCancelCheckpointHook (5): no-op outside subagent thread, no-op without file, no-op without cancel op, cancels on cancel op, first-writer-wins on natural completion

Test run

241 passed, 1 warning in 8.90s

All 241 unit tests pass including 10 new cancel-checkpoint tests.

Closes #3258

@greptile-apps

greptile-apps Bot commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds cooperative cancellation for subagents. The main changes are:

  • Writes cancellation records to each subagent control file.
  • Checks for cancellation before each thread-mode generation step.
  • Uses a distinct cancelled result status.
  • Treats cancelled agents as terminal in batch waits.
  • Adds unit and integration coverage for cancellation behavior.

Confidence Score: 5/5

This looks safe to merge.

  • Cancellation is stored with the intended status.
  • Batch waits now return when a cancelled agent finishes.
  • Result races preserve the first terminal outcome.
  • No blocking issues remain in the changed code.

Important Files Changed

Filename Overview
gptme/tools/subagent/api.py Writes cancellation operations and records explicit cancellation results across execution modes.
gptme/tools/subagent/hooks.py Adds a cooperative checkpoint that stops a subagent at the next step boundary.
gptme/tools/subagent/execution.py Allows current subagent identity to come from thread-local state or the subprocess environment.
gptme/tools/subagent/batch.py Recognizes cancelled results as terminal when waiting for any subagent.
tests/test_subagent_unit.py Adds coverage for control-file writes, cooperative checkpoints, races, and cancelled batch results.
tests/test_subagent_integration.py Updates cancellation lifecycle expectations to use the cancelled status.

Reviews (5): Last reviewed commit: "fix(tests): parse JSONL properly in canc..." | Re-trigger Greptile

Comment thread gptme/tools/subagent/api.py
Comment thread gptme/tools/subagent/api.py
@TimeToBuildBob

Copy link
Copy Markdown
Member Author

@greptileai review

@TimeToBuildBob

Copy link
Copy Markdown
Member Author

Addressed two P1 findings from the Greptile review (ea50be2):

1. Cancel status was "failure" instead of "cancelled"

subagent_cancel() defined cancelled_result = ReturnType("failure", ...), then called set_subagent_result_if_absent() before writing the cancel op. The cooperative checkpoint's later attempt to store ReturnType("cancelled", ...) always lost the first-writer-wins race — so callers (subagent_status, subagent_wait) saw every cooperative cancel as a plain failure. Fixed by using ReturnType("cancelled", ...) consistently.

2. Subprocess agents never observed the cancel op

get_current_agent_id() read only the thread-local, which is never set in subprocess agents (they receive their identity via GPTME_SUBAGENT_AGENT_ID env var). So the STEP_PRE checkpoint always returned None → early exit, never reading control.jsonl. Fixed by adding an os.environ.get("GPTME_SUBAGENT_AGENT_ID") fallback.

Test suite updated to assert "cancelled" status (not "failure") for all cancel paths. All 241 tests pass.

@TimeToBuildBob
TimeToBuildBob force-pushed the fix-3258-cancel-checkpoint branch from 80e8322 to 019e8f3 Compare July 15, 2026 17:04
@TimeToBuildBob

Copy link
Copy Markdown
Member Author

@greptileai review

Comment thread gptme/tools/subagent/api.py
Comment thread gptme/tools/subagent/api.py
@TimeToBuildBob

Copy link
Copy Markdown
Member Author

@greptileai review

@TimeToBuildBob

Copy link
Copy Markdown
Member Author

Fixed the remaining P1 finding from the latest Greptile review (84216ff):

cancelled not recognized as terminal in wait_any()

subagent_wait_any() had terminal_statuses = {"success", "failure", "clarification_needed"} — cancelled was missing. When a subagent was cancelled, _wait_one_notify() returned early without setting done_event, causing wait_any() to keep polling until another agent finished or the timeout expired. Added cancelled to the set and a regression test that verifies wait_any() returns immediately on a cancelled result.

The other three Greptile findings:

  • "Cancellation Remains A Failure" (line 1075) — false positive; the prior commit (865bd86) already stores ReturnType("cancelled", ...) not failure.
  • "Subprocess Cannot Read Cancel Operation" / "Checkpoint remains unreachable" — valid scope observation: the cooperative checkpoint is thread-local by design and subprocess cancel already works via SIGTERM. No regression introduced.

@codecov

codecov Bot commented Jul 15, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 50.44248% with 112 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
tests/test_subagent_unit.py 28.47% 107 Missing and 1 partial ⚠️
tests/test_subagent_integration.py 0.00% 2 Missing ⚠️
gptme/tools/subagent/hooks.py 96.29% 1 Missing ⚠️
tests/test_tools_subagent.py 0.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@TimeToBuildBob

Copy link
Copy Markdown
Member Author

All Greptile review findings addressed after 3 rounds. Summary:

Fixed (code changes):

  • "Cancellation Remains A Failure" — ReturnType("cancelled", ...) used consistently; checkpoint result is no longer shadowed
  • "Subprocess Cannot Read Cancel Operation" — os.environ.get("GPTME_SUBAGENT_AGENT_ID") fallback added to get_current_agent_id()
  • "Cancelled is not terminal" — cancelled added to terminal_statuses in subagent_wait_any() with regression test

Dismissed (design / false positive):

  • "Checkpoint remains unreachable" (subprocess) — the cooperative checkpoint is thread-local by design; subprocess agents already exit via SIGTERM within the 5s force-kill window. Adding the hook without the full subagent tool would add complexity for a path that already works.

CI: All checks pass; "Test without API keys" was still running at last check.

Greptile: 5/5 confidence, all threads resolved. Converged after 3 review rounds.

Implements the STEP_PRE cancel checkpoint described in issue #3258.
Cancel signals written to logdir/control.jsonl are detected before
each LLM generation step, allowing subagent_cancel() to actually stop
running thread-mode subagents without waiting for natural completion.

Changes:
- hooks.py: add _subagent_cancel_checkpoint() — fires at STEP_PRE,
  reads control.jsonl for a cancel op, yields a status message, writes
  'cancelled' result via set_subagent_result_if_absent (first-writer-wins),
  then raises SessionCompleteException to cleanly release the semaphore slot
- api.py: add _write_cancel_op() helper (flock-safe JSONL append) and call
  it from subagent_cancel() in both thread and subprocess modes
- __init__.py: register cancel_checkpoint hook at STEP_PRE priority 100

Two O(1) fast-path guards keep per-step overhead minimal:
  1. Thread-local check: no-op in parent thread (agent_id is None)
  2. stat() on control.jsonl: no-op when file doesn't exist

ACP mode retains existing cache-mark-only behavior (no control.jsonl write).

Closes #3258
Two bugs found by Greptile code review:

1. `cancelled_result` was `ReturnType("failure", ...)` — so `subagent_cancel()`
   stored "failure" before writing the control op, meaning the cooperative
   checkpoint's later `set_subagent_result_if_absent("cancelled", ...)` always
   lost the first-writer-wins race.  Callers saw the cancel as a failure.
   Fixed: use `ReturnType("cancelled", ...)` consistently.

2. `get_current_agent_id()` read only the thread-local, which is unset in
   subprocess agents (they receive identity via `GPTME_SUBAGENT_AGENT_ID` env).
   So the STEP_PRE checkpoint exited early for subprocess agents and they never
   observed the cancel op.  Fixed: fall back to `os.environ.get(...)`.

Update all tests that asserted the old "failure" cancel status to "cancelled".
…led'

subagent_cancel() now sets ReturnType('cancelled', ...) instead of 'failure'.
Two pre-existing tests still expected 'failure'; update them to match.
subagent_wait_any() skipped cancelled results because 'cancelled' was
not in terminal_statuses — the polling loop would keep waiting for
another agent to finish instead of returning the cancelled result.

Add 'cancelled' alongside 'success', 'failure', 'clarification_needed'
and a regression test that verifies wait_any() returns immediately on
a cancelled result.
@TimeToBuildBob
TimeToBuildBob force-pushed the fix-3258-cancel-checkpoint branch from 84216ff to dc4dafb Compare July 15, 2026 18:32
@TimeToBuildBob

Copy link
Copy Markdown
Member Author

Rebased onto master to resolve merge conflicts with #3262 (which implemented an overlapping cooperative cancel checkpoint).

Conflict resolution notes:

  • __init__.py: kept master's "control" hook (_subagent_control_hook) — both hooks solve the same problem; kept the one that's already in master to avoid registering two step.pre hooks that both drain/read the control file
  • api.py: kept master's append_control_op call (writes cancel op before subprocess termination) and used our updated docstring; carried forward the ReturnType("cancelled", ...) fix so subagent_cancel() sets "cancelled" not "failure" — this wins the first-writer-wins race before the hook fires
  • tests/test_subagent_unit.py: combined both import sets (master's SessionCompleteException for _subagent_control_hook tests + our _write_cancel_op for TestWriteCancelOp tests)

The unique contributions of this PR that weren't in #3262:

  1. ReturnType("cancelled", ...) in subagent_cancel() instead of "failure"
  2. "cancelled" added to wait_any terminal statuses (so subagent_wait_any returns immediately on cancel)

The test was attempting to parse the entire JSONL file as a single JSON object,
but control.jsonl is a JSON Lines format with one record per line. Fixed by
splitting on newlines and parsing the first line.

This fixes the test failures:
- test_cancel_thread_writes_control_file
- test_cancel_subprocess_writes_control_file

Both were failing with: json.decoder.JSONDecodeError: Extra data: line 2 column 1
@TimeToBuildBob

Copy link
Copy Markdown
Member Author

@greptileai review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(subagent): cooperative control checkpoint — make cancel actually stop thread-mode subagents

1 participant