Repository navigation
[vLLM][DRIVER] Segfault due to dlclose() of spirv_utils module with GC-invisible PyKernelArgObject references #7682
Description
Problem
Long-running vLLM workloads on Intel XPU with torch.compile enabled intermittently crash with:
Fatal Python error: Segmentation fault
Core dump analysis (symbolized via debuginfod.ubuntu.com) shows the crash happening inside the cyclic garbage collector while traversing a tuple:
Py_TYPE(ob=0x7f579f4fe500) <- object.h:220, invalid memory
_PyObject_IS_GC(obj)
visit_decref(op, parent=tuple@0x7f57684b33c0)
tupletraverse(o, visit, arg)
gc_collect_main -> subtract_refs -> visit_decref -> tupletraverse
Reproduction steps
Repeatedly running vLLM's CPU-offload correctness test reproduces the crash within ~10 runs on Intel XPU:
for i in $(seq 1 10); do
pytest -sv tests/basic_correctness/test_cpu_offload.py
done
Workaround
Setting VLLM_DISABLE_COMPILE_CACHE=1 makes the loop above pass:
for i in $(seq 1 10); do
VLLM_DISABLE_COMPILE_CACHE=1 pytest -sv tests/basic_correctness/test_cpu_offload.py
done
Possible fix
Remove the del/dlclose() and teardown from SpirvUtils in third_party/intel/backend/driver.py, so spirv_utils.so is never unloaded for the lifetime of the process — matching what the NVIDIA and AMD backends already do for their equivalent driver singletons, neither of which ever calls dlclose().
Why: spirv_utils is compiled exactly once per process today, so there's nothing to clean up by unloading it, and ctypes.PyDLL/CDLL never unload libraries on their own anyway — the dlclose() call is extra code specific to this backend. Removing it should be low-risk. ArchParser and ExtensionUtils are not part of this proposal and would keep their existing dlclose() behavior.
Environment details
==============================
System Info
==============================
OS : Ubuntu 24.04.4 LTS (x86_64)
GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version : Could not collect
CMake version : version 4.4.0
Libc version : glibc-2.39
==============================
PyTorch Info
==============================
PyTorch version : 2.13.0+xpu
Is debug build : False
CUDA used to build PyTorch : None
ROCM used to build PyTorch : N/A
XPU used to build PyTorch : 20260000
==============================
Python Environment
==============================
Python version : 3.12.3 (main, Jun 19 2026, 12:46:00) [GCC 13.3.0] (64-bit runtime)
Python platform : Linux-6.14.0-1009-intel-x86_64-with-glibc2.39
==============================
Intel XPU / GPU Info
==============================
Is XPU available : True
XPU runtime version : 20260000
Intel GPU models :
GPU 0: Intel(R) Arc(TM) Pro B60 Graphics
GPU 1: Intel(R) Arc(TM) Pro B60 Graphics
GPU 2: Intel(R) Arc(TM) Pro B60 Graphics
GPU 3: Intel(R) Arc(TM) Pro B60 Graphics
--Compile time--
oneAPI compiler version : Could not collect
SYCL compiler build : Could not collect
oneCCL version : 2022.0.0
--Runtime--
Intel Graphics Compiler (IGC): 2.34.4
Intel GMM (libigdgmm) : 22.10.0
Level Zero loader version : 1.28.2
Level Zero driver version : 26.18.38308.1-0
vLLM XPU kernels version : 0.1.12
==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] pyzmq==27.1.0
[pip3] sentence-transformers==5.3.0
[pip3] torch==2.13.0+xpu
[pip3] torchaudio==2.11.0+xpu
[pip3] torchcodec==0.15.0+cpu
[pip3] torchvision==0.28.0+xpu
[pip3] transformers==5.14.1
[pip3] triton-xpu==3.7.2
[conda] Could not collect
==============================
vLLM Info
==============================
ROCM Version : Could not collect
vLLM Version : 0.26.1rc1.dev167+g27f23df15.d20260731 (git sha: 27f23df15, date: 20260731)
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; XPU: Enabled
GPU Topology:
Could not collect
==============================
Environment Variables
==============================
VLLM_WORKER_MULTIPROC_METHOD=spawn
VLLM_TARGET_DEVICE=xpu
LD_LIBRARY_PATH=/tmp/ucx_install/lib:/opt/venv/lib:/usr/local/lib
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
Activity
[-][vLLM][DRIVER] Intermittent segfault during GC, likely related to spirv_utils.so being unloaded while still referenced[/-][+][vLLM][DRIVER] Segfault due to dlclose() of spirv_utils module with GC-invisible PyKernelArgObject references[/+]chaojun-zhang commentedon Aug 12, 2026
https://buildkite.com/vllm/intel-ci/builds/8664#019ff15f-5f8a-4346-a4b2-e670d9ee8971 another segfault issue
[XPU][DRIVER] Never unload `spirv_utils`: fixes GC segfault (#7682)
[XPU][DRIVER] Let the #7682 canary run without an XPU or a working torch
[XPU][DRIVER] Never unload `spirv_utils`: fixes GC segfault (#7682)
[XPU][DRIVER] Make the #7682 regression test cover Windows and torch-…
[XPU][DRIVER] Never unload `spirv_utils`: fixes GC segfault (intel#7682…