Steps to reproduce
- Create a fleet pinned to
g4-standard-6:
type: fleet
name: gcp-g4-fractional
nodes: 1
backends: [gcp]
regions: [us-central1]
instance_types: [g4-standard-6]
spot_policy: spot
resources:
gpu: RTXPRO6000:1
- Run a minimal CUDA task on that fleet:
type: task
name: gcp-g4-fractional-cuda
image: nvidia/cuda:13.0.3-base-ubuntu24.04
commands:
- ls -l /dev/nvidia*
- nvidia-smi
fleets: [gcp-g4-fractional]
creation_policy: reuse
resources:
gpu: RTXPRO6000:1
Actual behaviour
The plan reports g4-standard-6 as RTXPRO6000:96GB:1. Google documents this machine type as a fractional 1/8 GPU with 12GB of GPU memory.
The instance boots dstack-cuda-0-14, which uses the open NVIDIA kernel modules. The host kernel rejects the G4 vGPU:
NVRM: The NVIDIA vGPU 0000:00:03.0 (PCI ID: 10de:2bb5)
NVRM: installed in this system is not supported by open nvidia.ko.
nvidia 0000:00:03.0: probe with driver nvidia failed with error -1
NVRM: None of the NVIDIA devices were initialized.
The shim logs host nvidia-smi exit code 9 and detects no GPUs. The whole-instance job still starts without an NVIDIA Docker device request, so the task has no /dev/nvidia0 and nvidia-smi is not injected.
This also lets g4-standard-6 incorrectly satisfy requirements above its real 12GB GPU memory.
Expected behaviour
Until fractional G4 vGPUs are explicitly supported, dstack must not return or provision:
g4-standard-6 (1/8 GPU, 12GB)g4-standard-12 (1/4 GPU, 24GB)g4-standard-24 (1/2 GPU, 48GB)
Full-GPU G4 machine types must remain available. A control smoke on g4-standard-48 passed with the same dstack VM image and CUDA container: host driver 580.167.08, CUDA 13.0, and 97,887 MiB visible in the container.
dstack version
0.20.27 with gpuhunt 0.1.26
Additional information
The original G4 support PR documented g4-standard-48; the GA PR listed only g4-standard-48, g4-standard-96, g4-standard-192, and g4-standard-384. Fractional G4 support was not intended.
Suggested immediate fix:
- Filter
g4-standard-6, g4-standard-12, and g4-standard-24 from GCP offers in dstack, with a focused test that keeps g4-standard-48 supported.
- Fix gpuhunt so unsupported fractional shapes are not published as full
RTXPRO6000:96GB GPUs.
- Track actual fractional-vGPU support separately. It requires effective memory reporting, GCP's vGPU/GRID driver, and VM identity authentication; globally classifying all
RTXPRO6000 GPUs as proprietary-driver devices would be incorrect.
Google references:
Steps to reproduce
g4-standard-6:Actual behaviour
The plan reports
g4-standard-6asRTXPRO6000:96GB:1. Google documents this machine type as a fractional 1/8 GPU with 12GB of GPU memory.The instance boots
dstack-cuda-0-14, which uses the open NVIDIA kernel modules. The host kernel rejects the G4 vGPU:The shim logs host
nvidia-smiexit code 9 and detects no GPUs. The whole-instance job still starts without an NVIDIA Docker device request, so the task has no/dev/nvidia0andnvidia-smiis not injected.This also lets
g4-standard-6incorrectly satisfy requirements above its real 12GB GPU memory.Expected behaviour
Until fractional G4 vGPUs are explicitly supported, dstack must not return or provision:
g4-standard-6(1/8 GPU, 12GB)g4-standard-12(1/4 GPU, 24GB)g4-standard-24(1/2 GPU, 48GB)Full-GPU G4 machine types must remain available. A control smoke on
g4-standard-48passed with the same dstack VM image and CUDA container: host driver580.167.08, CUDA13.0, and 97,887 MiB visible in the container.dstack version
0.20.27 with gpuhunt 0.1.26
Additional information
The original G4 support PR documented
g4-standard-48; the GA PR listed onlyg4-standard-48,g4-standard-96,g4-standard-192, andg4-standard-384. Fractional G4 support was not intended.Suggested immediate fix:
g4-standard-6,g4-standard-12, andg4-standard-24from GCP offers in dstack, with a focused test that keepsg4-standard-48supported.RTXPRO6000:96GBGPUs.RTXPRO6000GPUs as proprietary-driver devices would be incorrect.Google references: