Summary
The fifo pipe-liveness watchdog (services/fifo_reader.py, _check_pipe_liveness) has one remaining blind spot in the burst→stall→settle family: when the burst, the forwarder stall, and the pane settling into its final frame all happen inside a single liveness interval (PIPE_LIVENESS_CHECK_INTERVAL_S, default 4s), the next check observes "FIFO delivered bytes since last check" → takes the healthy: re-baseline arm → and pins the new baseline to the post-stall settled frame. From then on the pane matches the baseline and the FIFO is silent, which is indistinguishable from a legitimately idle terminal, so no strike ever accumulates and the pipe is never re-armed — while the rolling buffer stays permanently stuck on stale mid-burst content and status detection stays wrong for the rest of the terminal's life.
The multi-interval variant of this was fixed by pinning the baseline across checks (the current docstring at fifo_reader.py:437-452 documents that case well — settle after an observed divergence now keeps accumulating strikes). The single-interval variant slips past because the re-baseline decision keys on "any bytes since last check", and the pre-stall burst satisfies it.
Credit
First observed by @reyph in #463 (the "stall during burst" scenario in their incident notes). Re-filed standalone since the PR has gone quiet.
Fix sketch
The re-baseline arm needs a stronger predicate than "bytes arrived since last check". Two options that compose: (a) only re-baseline when the pane content at check time is reachable from the FIFO-fed buffer tail (cheap suffix check — if the pane shows content the buffer never received, bytes-arrived is not proof of health); (b) re-baseline to the pane frame captured at the moment the last FIFO bytes were seen rather than at check time. Either keeps the idle-terminal false-positive guarantee (both still require FIFO-silent + divergence before re-arming).
Summary
The fifo pipe-liveness watchdog (
services/fifo_reader.py,_check_pipe_liveness) has one remaining blind spot in the burst→stall→settle family: when the burst, the forwarder stall, and the pane settling into its final frame all happen inside a single liveness interval (PIPE_LIVENESS_CHECK_INTERVAL_S, default 4s), the next check observes "FIFO delivered bytes since last check" → takes the healthy: re-baseline arm → and pins the new baseline to the post-stall settled frame. From then on the pane matches the baseline and the FIFO is silent, which is indistinguishable from a legitimately idle terminal, so no strike ever accumulates and the pipe is never re-armed — while the rolling buffer stays permanently stuck on stale mid-burst content and status detection stays wrong for the rest of the terminal's life.The multi-interval variant of this was fixed by pinning the baseline across checks (the current docstring at
fifo_reader.py:437-452documents that case well — settle after an observed divergence now keeps accumulating strikes). The single-interval variant slips past because the re-baseline decision keys on "any bytes since last check", and the pre-stall burst satisfies it.Credit
First observed by @reyph in #463 (the "stall during burst" scenario in their incident notes). Re-filed standalone since the PR has gone quiet.
Fix sketch
The re-baseline arm needs a stronger predicate than "bytes arrived since last check". Two options that compose: (a) only re-baseline when the pane content at check time is reachable from the FIFO-fed buffer tail (cheap suffix check — if the pane shows content the buffer never received, bytes-arrived is not proof of health); (b) re-baseline to the pane frame captured at the moment the last FIFO bytes were seen rather than at check time. Either keeps the idle-terminal false-positive guarantee (both still require FIFO-silent + divergence before re-arming).