Conversation
CutFromIOBufList() posts every message by reference, so the NIC has to read a small RPC from host memory before sending it. Request 236 bytes of inline data at QP creation, retrying without it if the device refuses, and inline a message that fits the granted size and comes entirely from the RDMA block pool. 236 bytes is the largest Send whose WQE fits the 256-byte BlueFlame buffer of current mlx5 NICs. Generated-by: Claude Code (Claude Opus 5.5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alxrxs
force-pushed
the
rdma-inline-small-messages
branch
from
October 2, 2026 12:56
91efa9b to
7da4d18
Compare
alxrxs
marked this pull request as ready for review
October 2, 2026 13:57
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The new inline-selection and QP-fallback paths need automated regression coverage.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Adds inline posting for small RDMA messages to reduce DMA-read latency.
Changes:
- Requests up to 236 bytes of inline QP capacity with fallback.
- Inlines eligible pool-backed messages while excluding user-registered memory.
| File | Description |
|---|---|
src/brpc/rdma/rdma_endpoint.h |
Stores granted inline capacity. |
src/brpc/rdma/rdma_endpoint.cpp |
Negotiates capacity and selects inline sends. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+907
to
+908
| if (in_pool && this_len <= _resource->max_inline_data) { | ||
| wr.send_flags |= IBV_SEND_INLINE; |
Comment on lines
+1157
to
+1160
| if (qp == nullptr) { | ||
| // The device may not support inline data, try again without it | ||
| attr.cap.max_inline_data = 0; | ||
| qp = IbvCreateQp(GetRdmaPd(), &attr); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: none
Problem Summary:
RdmaEndpoint::CutFromIOBufList() posts every message by reference, so for a small RPC request or response the NIC first has to DMA-read the data from host memory. The QP requests no inline data (MAX_INLINE_DATA was never used and was commented out in #2876).
What is changed and the side effects?
Changed:
236 bytes is the largest Send whose inlined WQE fits the 256-byte BlueFlame buffer of current mlx5 NICs. Larger messages are posted as before.
Side effects:
Performance effects: removes a host memory read before each small message is sent. In a separate ping-pong microbenchmark that sends the same small RDMA write with and without IBV_SEND_INLINE, inlining cut one-way latency by about 0.43 us on Xeon 8480C hosts with ConnectX-7 and 0.89 us on EPYC 7742 hosts with ConnectX-6. I haven't measured brpc itself. With the default rdma_max_sge the send WQE does not grow, but with a small rdma_max_sge it can.
Breaking backward compatibility: no.
Check List:
Tested on soft-RoCE (rxe) with the rdma_performance server and an echo client: attachments from 0 to 300 KB compared byte for byte, messages up to 236 bytes went inline and larger ones by reference, and the same again with the QP creation retry forced. libbrpc.a builds with WITH_RDMA=ON.
Generated with Claude Code