nixl backend issue for kv offloading #18198

First: your command as pasted never applied --hicache-storage-backend nixl

Look carefully at the backslashes:

  --enable-hierarchical-cache \
  --hicache-ratio 2
  --hicache-write-policy write_through
  --hicache-mem-layout page_first_direct
  --hicache-storage-backend nixl

Only the --enable-hierarchical-cache line is continued. The line without a trailing \ terminates the command, so the shell tries to execute --hicache-write-policy write_through as a program (command not found) and the server was actually launched with just ... --enable-hierarchical-cache --hicache-ratio 2. If that's how you ran it, NIXL was never selected and the warnings came from somewhere else. If it's a paste artifact, ignore this — but it's worth checking your shell history, because it silently changes which backend is in use.

Assuming the flags were applied, here's the actual failure mode.

What createXferReq: no specified or potential backend had the required registrations means in sglang

NIXL can only move memory it has a registration for on both ends. In HiCacheNixl the two ends are handled completely differently:

• Host (DRAM) side: pre-registered once, up front. register_mem_pool_host() decides zero-copy vs copy mode, then registers:

  # enable zero-copy automatically if mem layout is page_first or page_first_direct
  self.is_zero_copy = self.mem_pool_host.layout in ["page_first", "page_first_direct"]
  ...
  if self.is_zero_copy:
      self._pre_register_host(kv.data_ptr(), kv.numel() * kv.element_size(), "kv_buffer")
  else:
      self._bounce_set = self._alloc_registered(..., "bounce_set")
      self._bounce_get = self._alloc_registered(..., "bounce_get")

• Storage (file) side: registered per transfer in NixlRegistry.storage(), and every transfer goes through initialize_xfer(...). That call is what emits your message when the descriptors can't be matched to a backend.

So the error means: for at least one of the two descriptor sets, NIXL has no backend registration it can use.

The most likely cause: auto plugin selection picked something you don't have

This is the trap I'd check first. With no explicit plugin, sglang tries the file plugins in this priority order:

FILE_PLUGINS = ["3FS", "POSIX", "GDS_MT", "GDS"]
OBJ_PLUGINS = ["OBJ"]  # Based on Amazon S3 SDK
...
if self.plugin == "auto":
    for plugin in self.FILE_PLUGINS:
        if plugin in plugin_list:
            self.backend_name = plugin
            break

So if the 3FS plugin is present in your NIXL install, auto selects 3FS — not POSIX. If you don't actually have a 3FS filesystem mounted at the storage path, registration against it fails and you get exactly this message. Same story for GDS/GDS_MT if GPU Direct Storage isn't usable on your box.

Pin it explicitly:

export SGLANG_HICACHE_NIXL_BACKEND_PLUGIN=POSIX

and confirm which plugin actually got selected by grepping the startup log for the info-level line:

NixlBackendSelection.create_backend: backend_name <X> initparams {...} supported plugins [...]

That line tells you the chosen backend, the available plugin list, and the init params in one shot — it's the fastest way to see whether "who you think you're using" and "who you're actually using" differ.

Second: O_DIRECT is on by default and is a common source of this

def get_use_direct_io(self) -> bool:
    if "use_direct_io" in self.config:
        return bool(self.config["use_direct_io"])
    return envs.SGLANG_HICACHE_NIXL_USE_DIRECT_IO.get()   # default: True

With O_DIRECT on, the storage path must live on a filesystem that supports it, and buffers must be OS-page aligned. If --hicache-storage-backend points at something like an overlayfs/tmpfs container path, registration/transfer fails. Two things to try:

SGLANG_HICACHE_NIXL_USE_DIRECT_IO=0 ...     # rule O_DIRECT out cheaply

and set the storage directory explicitly to real local disk:

export SGLANG_HICACHE_NIXL_BACKEND_STORAGE_DIR=/mnt/nvme/hicache_nixl

(That env var exists: SGLANG_HICACHE_NIXL_BACKEND_STORAGE_DIR = EnvStr(None). When unset it falls back to the file_path argument, i.e. /tmp/hicache_storage by default — worth checking, since a container /tmp is often not a good O_DIRECT target and is also where you'd least want a large KV tier.)

Third: page_first_direct triggers zero-copy, which has a hard alignment requirement

Because your layout is page_first_direct, is_zero_copy is set True. Then the host kv_buffer is used directly as the NIXL DRAM region, which requires the base pointer and the per-page stride to be 4096-aligned. If they aren't, sglang falls back to copy mode — and it says so:

HiCacheNixl: O_DIRECT is active but the host kv_buffer is not OS-page-aligned (base or per-page stride). Falling back to copy mode for this pool.

Grep your logs for not OS-page-aligned and for HiCacheNixl: pre-registered host regions / registered logical anchor pool. Their presence or absence tells you which of the three paths (zero-copy / bounce-buffer / logical-anchor) you actually took — and the logical-anchor path matters here, because for the DeepSeek-V4-style LogicalHostPool there is no real KV buffer and sglang registers only small marker pages:

kv = getattr(mem_pool_host, "kv_buffer", None)
if kv is None:
    # DeepSeek V4 uses a LogicalHostPool as the KV anchor ... It has no actual KV bytes
    self.is_zero_copy = False
    self._logical_anchor = True

If your model path resolves to that anchor, the host side carries no KV bytes and the component pools do the work through the v2 APIs — a very different registration shape, and one where a component pool that failed to register produces your exact error.

Also worth checking

• --page-size 128 with HiCache: larger pages mean larger per-transfer regions; combined with a plugin that has a max-registration or alignment constraint this can be the thing that tips it over. For a quick bisect, drop to the default page size and see whether the warnings disappear — that separates "plugin/registration problem" from "this specific geometry."
• --hicache-ratio 2 sizes the host tier at 2× the device KV cache. Make sure the node actually has that much free DRAM per server on top of weights; if the tier is being allocated thin, failures under load look like transfer errors.
• --hicache-write-policy write_through: legal (write_back / write_through / write_through_selective), so that one's fine.

Suggested bisect order

1. Re-check the command actually passed --hicache-storage-backend nixl (the missing backslashes).
2. Pin SGLANG_HICACHE_NIXL_BACKEND_PLUGIN=POSIX, confirm via the create_backend log line.
3. Set SGLANG_HICACHE_NIXL_BACKEND_STORAGE_DIR to real local disk, and try SGLANG_HICACHE_NIXL_USE_DIRECT_IO=0.
4. Grep for the alignment / pre-registration log lines to learn which host path you're on.
5. Only then vary --page-size.

If you can post the create_backend: line, the NIXL plugin list, and whichever of the HiCacheNixl: registration lines appear, I can tell you which of the above it is rather than guessing.