Skip to content

Drop abandoned requests when draining ZeroMQ send queue (#68660) - #70338

Open
dwoz wants to merge 1 commit into
saltstack:3008.xfrom
dwoz:dwoz/fix/zmq-drop-abandoned-requests-3008x
Open

dwoz wants to merge 1 commit into
saltstack:3008.xfrom
dwoz:dwoz/fix/zmq-drop-abandoned-requests-3008x

Conversation

@dwoz

@dwoz dwoz commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Port PR #70283 (originally landed on 3006.x as commit 22fb248) to 3008.x. Also extend the fix to RequestClient._send_recv, which is the primary REQ path on 3008.x -- AsyncReqMessageClient is deprecated and unreferenced in salt/ on 3008.x, so the 3006.x-only patch would be a no-op there.

The invariant: send() enqueues (future, message) and arms _timeout_message. When the caller's timeout expires the future is completed, but its queue entry remains and keeps pinning the serialized payload until the drain loop reaches it. self._queue has no maxsize and a REQ socket permits one request/reply in flight, so under sustained load the enqueue rate outruns the drain rate and the queue grows without bound. _send_recv now drops a request whose future is already done, rather than spending a round trip on a reply nobody can receive.

Changes:

  • salt/transport/zeromq.py::AsyncReqMessageClient._send_recv -- straight port of the 3006.x drop-abandoned block.
  • salt/transport/zeromq.py::RequestClient._send_recv -- same block, placed after the 'future is None' shutdown-sentinel check and before the socket.poll/send path. This is what actually clears the leak on 3008.x.
  • tests/pytests/unit/transport/test_zeromq.py -- three tests from the 3006.x change adapted for 3008.x (salt.ext.tornado unvendored, so the refs are rewritten to plain tornado; the existing test_client_send_recv_on_cancelled_error keeps 3008.x's newer RequestClient-based body). Plus one new test, test_request_client_send_recv_drops_abandoned_request, that asserts the invariant against RequestClient's async _send_recv(socket, queue) API.
  • changelog/68660.fixed.md -- from the cherry-pick.

)

Port PR saltstack#70283 (originally landed on 3006.x as commit 22fb248) to
3008.x. Also extend the fix to RequestClient._send_recv, which is the
primary REQ path on 3008.x -- AsyncReqMessageClient is deprecated and
unreferenced in salt/ on 3008.x, so the 3006.x-only patch would be a
no-op there.

The invariant: send() enqueues (future, message) and arms
_timeout_message. When the caller's timeout expires the future is
completed, but its queue entry remains and keeps pinning the serialized
payload until the drain loop reaches it. self._queue has no maxsize and
a REQ socket permits one request/reply in flight, so under sustained
load the enqueue rate outruns the drain rate and the queue grows
without bound. _send_recv now drops a request whose future is already
done, rather than spending a round trip on a reply nobody can receive.

Changes:
- salt/transport/zeromq.py::AsyncReqMessageClient._send_recv -- straight
  port of the 3006.x drop-abandoned block.
- salt/transport/zeromq.py::RequestClient._send_recv -- same block,
  placed after the 'future is None' shutdown-sentinel check and before
  the socket.poll/send path. This is what actually clears the leak on
  3008.x.
- tests/pytests/unit/transport/test_zeromq.py -- three tests from the
  3006.x change adapted for 3008.x (salt.ext.tornado unvendored, so the
  refs are rewritten to plain tornado; the existing
  test_client_send_recv_on_cancelled_error keeps 3008.x's newer
  RequestClient-based body). Plus one new test,
  test_request_client_send_recv_drops_abandoned_request, that asserts
  the invariant against RequestClient's async _send_recv(socket, queue)
  API.
- changelog/68660.fixed.md -- from the cherry-pick.

This branch was successfully deployed

1 active deployment
ci — f347080e Deployed Sep 30, 2026 by dwoz via Build Source Packages / macOS (x86_64) #27292
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:full Run the full test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants