Skip to content

Proposal: support LeRobot's ACT policy #37

Closed

Description

@ravediamond

ACT is probably the most common policy on SO-100/SO-101 arms, and it isn't in vla.cpp yet. I'd like to add it, but since it isn't a VLA (no language input), I wanted to check it's in scope first.

Why it could be worth it: PyTorch ACT is already fast on a Jetson (~59 ms per 100-action chunk on my Orin NX), but it still needs a CUDA PyTorch build for the board. On CPU-only boards like the one in #31, vla.cpp would be the lighter option.

What I have in mind, following "Adding an architecture" in CONTRIBUTING:

  • scripts/convert_act_to_gguf.py for LeRobot ACT checkpoints: ResNet18 with its batch norms folded into the convs, the normalizer stats from the processor files, and no VAE encoder (it's training-only, so the latent is zero at inference). That leaves ~34M of the 52M parameters.
  • src/models/act.cpp: ResNet18 on the conv path Octo uses, the 4-layer encoder, and the 1-layer decoder with 100 learned queries, similar to TurboVLA's ACT decoder but post-norm. No flow-matching steps.
  • Checked against PyTorch with vla_predict_check, then on my SO-101. I have an ACT checkpoint at 9/10 on the same task as the SmolVLA run in Deployment pipeline for SmolVLA on a real SO-101 arm (Qualcomm Dragonwing IQ-9075, ARM CPU) #31.

Would you take this? And if so, should --arch act in the LeRobot client fork come with it?

Activity

  1. khanhnd61-vr commented on Oct 5, 2026

    @khanhnd61-vr
    Collaborator

    Hi @ravediamond ,

    The plan is absolutely fine to me. Thanks for raising the the topic. Please open the PR when your implementation is ready. We will review and merge it.

    Feel free to add act option for --arch.

    One thing I would like to add on top of your development suggestion: when checking against PyTorch with vla_predict_check, could you also check with the weight of lerobot lerobot/act_aloha_sim_transfer_cube_human along side with your SO-101 checkpoint?

    Thanks for your contribution to the project!

  2. ravediamond commented on Oct 6, 2026

    @ravediamond
    ContributorAuthor

    Thanks! Will do. I'll check act_aloha_sim_transfer_cube_human against PyTorch too, alongside my SO-101 checkpoint, and add act to --arch in the client. I'll open the PR when it's ready.

  3. ravediamond commented on Oct 19, 2026

    @ravediamond
    ContributorAuthor

    Done in #38, closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions