vla.cpp

An efficient C++ inference engine for Vision-Language-Action (VLA) models, built on top of llama.cpp. It brings today's open VLA policies - SmolVLA, π0, BitVLA, Evo-1, and GR00T N1.5/1.6/1.7 and more - under one runtime, packaging each as a single self-contained GGUF that needs no Python or PyTorch at inference time. The binary can drive robots across CPU, Apple Silicon, or CUDA, scaling from consumer GPUs down to the Jetson-class boards.

• CMake ≥ 3.22
• A C++17 compiler (GCC 11+ or Clang 14+)
• CUDA 12.x (optional - required only for GPU builds)
• libzmq3-dev, libprotobuf-dev, protobuf-compiler

sudo apt-get install -y libzmq3-dev libprotobuf-dev protobuf-compiler

Identify your machine CUDA architecture:

Configure and build the source:

# Fetch llama.cpp at pinned tag and apply local patch
bash ./patches/patch.sh

# CPU build:
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

# CUDA build (set CMAKE_CUDA_ARCHITECTURES for your GPU):
cmake -B build \
    -DGGML_CUDA=ON \
    -DGGML_CUDA_GRAPHS=ON \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_CUDA_ARCHITECTURES=$CUDA_ARCHITECTURE
cmake --build build -j$(nproc)

If system cannot detect CUDA, declare CUDA explicitly in environment variables

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

On Apple Silicon (e.g. Mac Mini M4), Metal is enabled by default and runs both the transformer and vision tower on the GPU. See docs/backend/metal.md for building vla.cpp on macOS.

The eval scaffold under eval/ supports two simulators end-to-end. Each setup script bootstraps an isolated Python 3.10 uv venv next to itself and clones the upstream sim repo. Both require uv on PATH.

bash eval/sim/libero/setup_libero.sh

Clones LIBERO into eval/sim/libero/LIBERO/, creates eval/sim/libero/libero_uv/.venv/, and pins compatible versions of torch, lerobot, transformers, and gymnasium.

bash eval/sim/simpler/setup_SimplerEnv.sh

Clones SimplerEnv (and its nested ManiSkill2_real2sim) into eval/sim/simpler/SimplerEnv/, creates eval/sim/simpler/simpler_uv/.venv/, and pins gymnasium 0.29.1, numpy 1.26.4, transformers 4.51.3, plus the ManiSkill2 + SimplerEnv editable installs.

vla-server loads the model once at startup and answers ZeroMQ REQ/REP requests synchronously.

# SmolVLA / π0 (mmproj + ckpt):
./build/vla-server "$VLA_MMPROJ" "$VLA_GGUF"

# Evo-1 / BitVLA / GR00T-N1.{5,6,7} (vision baked into the ckpt - omit mmproj):
./build/vla-server "$VLA_GGUF"

When ready, the server prints:

vla-server: bound to tcp://*:5555. ready.

Bound address and port can be configured by --bind flag. Stop server with Ctrl-C.

eval/client/ ships an end-to-end LIBERO benchmark runner that drives vla-server directly over the protobuf protocol. Make sure the LIBERO venv from Install simulators is set up first.

With vla-server already running:

source eval/sim/libero/libero_uv/.venv/bin/activate
python eval/client/run_sim_client_direct.py \
    --task libero_object --task-id 0 --n-episodes 1 \
    --output-dir /tmp/libero_outputs \
    --arch "$VLA_ARCH"

Note:

• Use proper --arch flag (see Models) to match the GGUF that vla-server is serving.
• Pi0 uses the gated google/paligemma-3b-pt-224 tokenizer (huggingface-cli login + accept the licence, or point --tokenizer at a local copy)
• The GR00T arches need --stats-json <ckpt>/dataset_statistics.json (action/state un-normalisation) and an embodiment selected server-side via VLA_GR00T_EMBODIMENT (new_embodiment for N1.5, libero_panda for N1.6, libero_sim for N1.7), with VLA_GR00T_BF16_WEIGHTS=1 to fit the 8 GB card.

So far only GR00T-N1.6 is wired (the gr00t-n1d6-bridge checkpoint with the oxe_widowx embodiment). Start vla-server on port 5566 with oxe_widowx embodiment:

VLA_GR00T_BF16_WEIGHTS=1 VLA_GR00T_EMBODIMENT=oxe_widowx \
    ./build/vla-server "$GR00T_N1D6_GGUF"

Then drive it from the SimplerEnv venv (set up via Install simulators):

source eval/sim/simpler/simpler_uv/.venv/bin/activate
python eval/client/run_simpler_client_direct.py \
    --arch gr00t_n1_6 \
    --task-id oxe_widowx/widowx_spoon_on_towel --n-episodes 1 \
    --embodiment oxe_widowx --image-size 252 \
    --stats-json "$VLA_STATS_JSON"

$VLA_STATS_JSON is the statistics.json shipped beside the bridge GGUF. The default 224-px GGUF mis-localises on WidowX (≈20% success) - the 252-px build is required.

Each model ships a combined VLA GGUF (LM + action expert + dataset stats + arch config) and, where applicable, a matching mmproj GGUF (vision tower). SmolVLA, π0 and π0.5 ship a separate mmproj; BitVLA, Evo-1, VLA-Adapter and OpenVLA-OFT bake their vision tower into the combined GGUF, so no mmproj file is needed.

vla.cpp also supports gr00t_n1_5, gr00t_n1_6, evo1, vla_adapter and pi05. If you would rather convert a HuggingFace safetensors checkpoint yourself, scripts/ provides per-arch GGUF converters. Set up a venv for converter by:

# Assume third_party/llama.cpp has been cloned and patched
python3 -m venv .venv-converter
source .venv-converter/bin/activate
pip install -r third_party/llama.cpp/requirements/requirements-convert_hf_to_gguf.txt
pip install safetensors

Then run any of the per-arch converters (--help for the full flag list):

python scripts/convert_smolvla_to_gguf.py \
    --ckpt /path/to/smolvla-libero \
    --out  /path/to/smolvla-libero-bf16.gguf

Full libero_object sweep - all 10 tasks × 20 episodes (200 episodes per arch), run via vla-server + eval/client/run_sim_client_direct.py across three deployment targets: an RTX 3060 (sm_86), an NVIDIA Jetson AGX Orin (sm_87, Jetson-class deployment hardware), and an NVIDIA Jetson Orin Nano (8 GB) (sm_87, the cheapest Jetson and the project's primary deployment target). On the Orin Nano, the 8 GB budget forces the ~6 GB servers (gr00t_n1_5, pi0) to run split - server on the Nano, client (sim) on the RTX 3060 - while lighter models run co-located; GR00T-N1.6 and GR00T-N1.7 could not be loaded on 8 GB and are omitted there.

Columns report client/call (ms) and Peak RAM (MiB) for each hardware target.

Support matrix of models (rows) against platforms (columns). Legend: Y = supported (released and benchmarked), ~ = in progress, - = planned.

Looking ahead, we will support more models, more platforms, and continue to optimize the framework.

• Khanh Dang Nguyen
• Hung Thinh Ho
• Chinh Truong Nguyen
• An Thai Le

Licensed under the Apache License, Version 2.0.

Supported VLA models:

• SmolVLA - Hugging Face LeRobot team.
• π0,π0.5 - Physical Intelligence.
• BitVLA - Hongyu Wang et al.
• Evo-1 - Tao Lin et al.
• VLA-Adapter - Yihao Wang et al.
• OpenVLA-OFT - Moo Jin Kim et al.
• GR00T N1.x - NVIDIA Isaac.

Behavioural evaluation is built on:

• llama.cpp - LLM inference engine in C/C++.
• LIBERO - the lifelong-robot-learning benchmark suite our success-rate sweeps run on.
• SimplerEnv - the second simulator wired through the eval scaffold.