Skip to content

Uplink: a chat turn whose final response never arrives keeps the connection's turn lock forever #2029

Description

@MastaP

Summary

The local uplink allows one active chat turn per connection (and per principal and session): Subscription::begin_turn in crates/astrid-uplink/src/native/egress.rs. The lock is released in only two cases:

  • a final chat response for that session passes through the egress registry (completed_chat_session);
  • the connection drops (Drop for Subscription).

There is no timeout, and nothing releases the lock when the agent capsule can no longer send a final response, for example when the capsules are reloaded (ReloadCapsules) while a turn is in flight, or the agent-loop capsule crashes. Every later prompt on that connection is then refused, and the client is not told (#2030). The chat stays stuck until the client reconnects.

Observed on astrid 6de3fd3; the code is unchanged on main at f3f0728.

Observed

  1. A turn stalled in the agent loop (a model stream lost its terminal event, Streaming turns stall: the capsule dispatch queue (64) drops llm.v1.stream events, including the terminal one #2031).
  2. Capsules were reloaded with ReloadCapsules while that turn was in flight. The agent loop restarted without the turn, so its phase timeout never fired and no final response was ever published.
  3. Eight minutes later, a new prompt on the same astrid chat connection:
WARN astrid_uplink::native: dropped local uplink message security_event=true principal=default reason=this principal or connection already has an active turn

The chat showed "thinking" indefinitely.

Steps to reproduce

Unit level, with the same setup as cancel_turn_is_forwarded_only_by_the_active_connection in crates/astrid-uplink/src/native/tests.rs:

  1. Install the egress registry and subscribe one connection.
  2. Send a chat request (process_inbound begins the turn).
  3. Publish no final chat response for that session.
  4. Send a second chat request on the same connection: it is refused with "this principal or connection already has an active turn", and stays refused for as long as the connection lives.

End to end:

  1. astrid chat, send a prompt.
  2. While the turn is still running, reload the capsules (ReloadCapsules), or use an agent capsule that never answers.
  3. Send another prompt: it is dropped, and the chat never completes.

Expected

The turn lock is released when the turn can no longer complete. Possible triggers:

  • the capsule handling the turn is reloaded or unloaded;
  • a turn-level timeout in the uplink;
  • an explicit cancel. Today cancel is only forwarded to the agent loop, which may no longer know the turn.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions