Why Is It So Hard to Run LeRobot on NVIDIA Jetson Orin? (And How I Fixed a 50+ Hour Build Wall)

If you have ever tried running a modern PyTorch project on an NVIDIA Jetson board (ARM aarch64 architecture), you have almost certainly hit this exact wall: there is no official, up-to-date PyTorch wheel compiled with CUDA support for Jetson’s specific hardware combination.

You need a wheel built for aarch64 + your specific JetPack/CUDA version + your project’s required Python version. Neither NVIDIA’s official PyTorch redist distribution nor the community indices (pypi.jetson-ai-lab.io) carry it.

For LeRobot(Hugging Face's open-source library for real-world robotics, imitation learning, and Vision-Language-Action models), this isn't just a minor setup headache. LeRobot enforces strict minimum requirements: Python ≥3.12 and torchcodec ≥0.11.0 (PyTorch’s native library for GPU-accelerated video frame decoding).

Attempting a naive pip install lerobot on a Jetson Orin fails silently with vague ABI or import errors, masking the real culprit.

This post is about what it actually takes to bridge that gap. Not a high-level summary, but the dirty, low-level engineering traps you hit when compiling a cutting-edge deep learning stack from source on an embedded ARM device. The end result is lerobot-jetson --> a complete, prebuilt Docker image on GHCR. But the real value lies in understanding everything underneath it (even if you never use it).

[ Ubuntu 22.04 (JetPack 6.2 / CUDA 12.6) ]
       │
       ├──> [ Custom FFmpeg + dav1d ] (NVDEC hardware video decode)
       │           │
       ├──> [ PyTorch 2.11 (cp312) ] (Source build: TORCH_CUDA_ARCH_LIST=8.7)
       │           │
       │           ├──> [ torchcodec 0.11.0 ] (GPU tensor video decoding)
       │           └──> [ torchvision 0.26.0 ] (Matched C++ ABI)
       │
       └──> [ LeRobot Stack ] (Python 3.12+ / SmolVLA / ACT)

Glossary: Embedded ARM & Video Decoders

Before digging into the dependency graph, here are a few niche platform terms you will encounter:

• JetPack: NVIDIA’s custom Ubuntu Board Support Package (BSP) containing Linux drivers, CUDA, cuDNN, and low-level Jetson hardware acceleration APIs.
• aarch64: The 64-bit ARM CPU architecture powering Jetson boards. Pre-compiled x86_64 wheels on PyPI are binary-incompatible with this instruction set.
• Compute Capability 8.7 (TORCH_CUDA_ARCH_LIST=8.7): The Streaming Multiprocessor (SM) target architecture specific to NVIDIA Orin (Ampere architecture) GPUs.
• NVDEC vs. NVENC: Dedicated on-chip Hardware ASICs inside the Orin GPU chip. NVDEC handles hardware-accelerated video decoding (crucial for streaming multi-camera streams into policy networks), while NVENC handles video encoding.
• torchcodec: PyTorch’s modern framework designed to decode video files directly into PyTorch CUDA tensors without CPU bottlenecks.
• C++ ABI (Application Binary Interface): The low-level interface rule set governing C++ binary compatibility. If PyTorch and extensions like torchvision are compiled with differing ABI flags, they crash at runtime even if they pass import checks.

The Actual Blocker: Python 3.12 & The Dependency Gap

LeRobot's pyproject.toml mandates Python ≥3.12. Checking the standard channels for a Jetson-compatible wheel reveals why standard installs break:

1. NVIDIA PyTorch Redist: NVIDIA's official PyTorch wheels for JetPack stop at older CPython versions (e.g., Python 3.10).
2. The Community Index (pypi.jetson-ai-lab.io): Carries PyTorch 2.11, but explicitly targets CPython 3.10 (cp310). Its Torchcodec support stops at 0.10.0, well below LeRobot's requirement floor.

Before touching downstream libraries, there is no CUDA-enabled PyTorch wheel for CPython 3.12 (cp312) on Jetson anywhere. That is the root cause: you must compile PyTorch, and every C++ extension running on top of it—from source for this exact platform.

Why not jetson-containers?

The historical standard for running Deep Learning based containers on Jetson is dusty-nv/jetson-containers. However, relying on it for modern LeRobot stacks presents maintainability challenges: its maintained package definitions lagged behind upstream shifts. Torchcodec is omitted, package dependencies like pi0 were renamed to pi upstream without updates, and legacy CLI entry points broke. Depending on state container definitions means endless debugging errors to try to make it works.

The 5 Custom Builds (And Why Each Is Mandatory)

To achieve a working stack, five distinct components must be compiled from source in a strict, sequential order (if you pass the PyTorch, you have basically passed the biggest hurdle):

Six Silent Build Traps (And How to Fix Them)

These are not trivial typos; they are subtle, failure-prone behaviors that consume hours of debugging time on embedded hardware.

1. PyTorch auto-enables NVSHMEM and fails at the final link step

Setting USE_DISTRIBUTED=1 causes PyTorch’s CMake build system to auto-detect the nvidia-nvshmem pip package. It silently enables NVSHMEM (NVIDIA’s multi-node shared-memory framework), a feature completely irrelevant on a single-board computer (but super useful for clusters). The build progresses for four hours, only to fail at the final linking stage.

• Fix: Explicitly pass USE_NVSHMEM=0 in the build environment flags.

2. Disabling flash-attention fails if CMake cache persists

Flash-attention’s backward CUDA kernels use heavy C++ template expansions. Parallel compilation threads can consume upwards of 1.5GB RAM per worker, causing Out-Of-Memory (OOM) errors on a 16GB Orin NX. Passing USE_FLASH_ATTENTION=0 to setup.py bdist_wheel fails to apply if build/CMakeCache.txt was generated during a previous attempt. The cached setting overrides your command-line flag.

• Fix: Clear the build directory, or force configuration directly inside the build directory (of course this make the build painfully slower):

cmake -DUSE_FLASH_ATTENTION=OFF .

3. Disabling torch.distributed breaks single-GPU execution downstream

To avoid multi-GPU overhead, it seems logical to set USE_DISTRIBUTED=0. However, LeRobot’s core training and evaluation loops unconditionally call initialization checks like torch.distributed.is_initialized(). When built with USE_DISTRIBUTED=0, PyTorch generates a stub module that omits these functions entirely, causing lerobot-train to crash instantly with an AttributeError.

• Fix: Keep USE_DISTRIBUTED=1, but pair it with the lightweight CPU-based Gloo backend. You do not need full NCCL multi-GPU drivers to satisfy the process initialization API.

4. torchvision installs cleanly, imports cleanly, but crashes at runtime

Installing torchvision via standard pip install completes without error and passes basic import torchvision tests. However, calling a compiled C++ operator during model evaluation throws a fatal ABI runtime error:

RuntimeError: operator torchvision::nms does not exist

The prebuilt PyPI binary was compiled against a different PyTorch C++ ABI toolchain.

• Fix: Rebuild torchvision from source directly against your custom-built PyTorch package.

5. FFmpeg header layout drift breaks NVENC builds

FFmpeg’s vendored NVENC source files rely on explicit struct layouts from NVIDIA's Video Codec SDK headers. If headers pull a version that splits a struct field (e.g., changing countingType into countingTypeLSB and countingTypeMSB), compilation breaks. Furthermore, running ./configure on FFmpeg auto-enables NVENC whenever headers are detected.

• Fix: Pass explicit disable flags to the configuration step. Because LeRobot only needs hardware decoding (NVDEC), you can safely bypass encoder header mismatches:

./configure --enable-nvdec --disable-nvenc ...

6. Silent shell script failures mask compilation errors

Upstream build wrappers frequently use conditional fallback chains:

# Bad pattern found in third-party install scripts
try_prebuilt_binary || compile_from_source || echo "FAILED to build"

Without an explicit exit 1 following the final warning, the shell script exits with status code 0. Docker accepts the RUN step as successful, commits the image layer, and conceals the underlying build failure until deep inside runtime execution.

• Fix: Audit downstream scripts to enforce set -e or add strict error checking after fallback failures.

Breakdown of Build Execution Time

Compiling this entire stack from scratch on a Jetson Orin NX (16GB) requires 5 hours and 33 minutes.

[==================================================] PyTorch Compile: 5h 06m (92%)
[==                                                ] FFmpeg, dav1d, torchcodec, torchvision, LeRobot: 27m (8%)

PyTorch compilation represents 92% of the total time budget. Packaging support tools, FFmpeg binaries, and vision extensions finish in under 30 minutes combined.

Verification & Prebuilt Container

The resulting container was validated on physical hardware (Jetson Orin NX 16GB running JetPack 6.2 / CUDA 12.6):

import torch
import torchcodec
import lerobot
from lerobot.common.policies.smolvla.modeling_smolvla import SmolVLAPolicy

# 1. Verify CUDA acceleration
assert torch.cuda.is_available() is True
print(f"Device: {torch.cuda.get_device_name(0)}")

# 2. Verify GPU Video Decoding via NVDEC
decoder = torchcodec.decoders.VideoDecoder("sample_stream.mp4", device="cuda")
frame = decoder[0]
assert frame.is_cuda is True  # Hardware NVDEC decode directly to GPU memory

# 3. Verify Distributed Stub Initializer
assert torch.distributed.is_available() is True

# 4. Verify End-to-End VLA Model Loading
policy = SmolVLAPolicy.from_pretrained("lerobot/smolvla-base")
print("SmolVLA successfully loaded on Jetson Orin!")

To avoid repeating this compilation process, the finalized image is available via the GitHub Container Registry:

docker pull ghcr.io/ravediamond/lerobot-jetson:latest

docker run --runtime nvidia -it \
  -e HF_TOKEN=<your-hf-token> \
  ghcr.io/ravediamond/lerobot-jetson:latest

The full build configuration, Dockerfiles, and technical logs are maintained in the repository: ravediamond/lerobot-jetson.

Conclusion: Moving Beyond Green-Field Demos

It is easy to watch a demo of a Vision-Language-Action (VLA) model running smoothly and forget the hardware used and the stack underneath it. But edge robotics does not live in Cloud. The moment you bring modern AI frameworks like LeRobot onto physical hardware like the Jetson Orin, you are forced to reconcile cutting-edge Python dependencies with real-world C++ ABIs, hardware decoders, and architecture-specific CUDA targets.

Building lerobot-jetson wasn't just about creating a Docker image; it was about getting my hands dirty and understanding what lives underneath the abstraction layer. It’s an appreciation for the unglamorous low-level plumbing that actually makes edge AI possible.

The repository and container are published so you don't have to sink a week into solving these build traps yourself:

• Prebuilt Container: docker pull ghcr.io/ravediamond/lerobot-jetson:latest
• Repository & Build Logs: ravediamond/lerobot-jetson

I plan to keep this updated as new LeRobot releases drop. If you're running LeRobot on an Orin setup, give the image a spin, star the repo, and open a PR or issue if you run into any hardware-specific edge cases!