Bounded the memory a graphql-transport-ws client can make the server hold - #631

Problem

A graphql-transport-ws client could make the server hold memory without limit, per connection, in three ways:

• Oversized messages. The reader assembled a message of any size.
• Unbounded inbox. The inbox from the reader to the control loop had no bound, so messages sent faster than they were handled, for example while a query executed, queued without limit.
• Unbounded outbound queue. It stayed unbounded, and the server never waited for the client to read. A client that sent pings and read nothing made the connection hold every pong: an independent review measured 312 MB in under half a second.

Changes

• Message size (breaking).
  • GraphQLTransportWSOptions.MaxReceiveMessageSize is 4 MiB by default. A message over it closes the connection with 1009 (MessageTooBig) as soon as one byte beyond the limit is received, without buffering more of it.
  • A value that is not positive fails when the middleware is created, instead of aborting every accepted connection.
• Inbox.
  • At most 16 client messages wait for the control loop, with BoundedChannelFullMode.Wait. While it is full, the reader does not read the socket, so TCP flow control slows the client down.
  • The messages wait as their bytes and are deserialized when handled, because a parsed message of dense JSON takes about ten times its size.
• Outbound.
  • Messages are serialized when they are queued and counted in bytes.
  • Once more than 16 MiB would wait for a client that does not read, the pending send is cancelled, which aborts the socket.
  • A single larger message is still sent when nothing else waits.
• No deadlocks. Producers never wait for room:
  • workers report their end through a queue of their own, which the control loop reads before every client message;
  • the inbox is completed once the control loop stops, which fails a write the reader waits on.

Tests

The tests are xUnit, in AspNetCore/WebSocketConnectionTests.fs:

• messages at the limit, over it, and over the default limit, including connection_init;
• the reader stopping while the inbox is full;
• the connection ending while the reader waits;
• the ids of subscriptions that ended while the inbox was full being free again;
• a client that does not read being aborted;
• the byte limit of the sender queue;
• the startup check.

The new tests fail when the behaviour is removed. The WebSocket tests passed 5 of 5 repeated runs.

An independent review ran a real Kestrel host and covered:

• limits exactly at and over the size, across frames and split characters;
• backpressure and close codes;
• host shutdown, client close and client abort with a full inbox.

It found no deadlock, hang or leaked worker.

Verification:

• Unit tests: the whole project passes, 782 tests with 5 skipped.
• FAKE: the full pipeline passes, including the integration tests.

Left for later

These predate this PR:

• Ping handler: a custom ping handler and AsyncExecute get no cancellation token, so a hanging one delays shutdown.
• Id reuse: a worker sends complete before it reports its end, so a client reusing an id at once could still, rarely, get 4409.

🤖 Generated with Claude Code