RL locomotion steering vectors

This project tests whether adding a constant vector to a frozen reinforcement-learning policy's hidden activations can produce useful changes in locomotion. The first implementation collects unsteered rollouts, extracts contrasting activation means, selects intervention strengths on validation episodes, and checks them on fresh paired episodes. It uses no behavioral cloning and does not update the pretrained policy's weights.

The scientific objective is a useful, repeatable causal effect, rather than a particular speed change. HalfCheetah candidates concern speed, torso height, and action effort; Ant candidates concern lateral movement and turning. A nearly constant behavior is an extraction diagnostic, not proof that hidden-state steering is impossible. See the experiment protocol for the contrast rules, controls, usefulness thresholds, and limitations. The experiment register records measured progress; planned capabilities are not results.

Adding the saved 256-dimensional lateral vector at strength −0.1 to the frozen Ant actor's first ReLU changes its ground-plane trajectory. On 30 fresh paired confirmation episodes, mean world-Y velocity increases by 0.402 m/s (paired 95% bootstrap interval [0.308, 0.491]), with 96.83% of mean forward speed retained. These are equal-episode summaries of available post-onset motion, with a zero-summary convention for an episode ending before onset. The saved confirmation results and within-project recomputation support these numbers. This is sideways displacement relative to the world's axes; it can involve a changed heading and does not establish body-relative sidestepping or jumping.

The within-project causal/code audit reproduced the vector from fitting data, checked split separation, reconstructed SAC actions, and replayed the recorded physics. It found a constant hidden-activation addition followed by ordinary policy inference, with the original reward and environment. No scripted sideways motor rule or reward modification was found in the inspected implementation; the audit's exact scope and limitations are recorded with its evidence.

The effect is only partly specific to this vector. On the same confirmation seeds: Each control was confirmed at its own validation-selected strength:

• Norm-matched random directions shifted world-Y velocity by +0.15 to +0.25 m/s. The primary exceeded each of these, and the constant action bias, by 0.15–0.27 m/s. Only one random direction ran at the primary's −0.1; it trailed by 0.15 m/s [0.08, 0.22]. The other two ran at −0.05.
• A label-shuffled direction at −0.2 shifted it by +0.89 m/s, with 9/30 failures. That direction has cosine 0.60 with the primary, so it is not a label-free control.
• The world-Y specificity audit finds the effect is not specific to the lateral label. In validation, the turning vector, fitted on yaw-rate labels, moves world Y more. About a quarter of the selected shift appears at either sign, but that share is poorly determined. On average the direction of travel rotates +3.84° through a bounded heading offset, not a continuing turn. Reproduce it with uv run --no-sync python scripts/audit_ant005_specificity.py.
• The path-mechanism audit confirms the effect over the steps both arms recorded: +0.392 m/s [0.299, 0.477]. The torso turns +5.23°, more than the path, so the gait is not rigidly rotated. A generic drift toward +Y is at most marginal. Reproduce it with uv run --no-sync python scripts/audit_ant005_path_mechanism.py.

Independent audit with fresh seeds. An independent audit (PDF) did not use the earlier audits. It rebuilt the evidence chain from raw episodes and found no implementation, pairing, leakage or reporting error. It then ran a pre-registered test on 20 fresh resets:

• Replicated. The effect is +0.462 m/s [0.343, 0.587], with 19 of 20 pairs positive. Failures rise from 0/20 to 2/20, beyond the five-point quality allowance. These seeds are not the formal replication split, and they include the run's convention-reserved application cohort.
• Borderline against random directions. Against 99 norm-matched random directions, its sign-dependent response ranks at p = 0.05, exactly at the threshold.
• How it works. The vector is a perceived-heading direction: it tells the policy it faces slightly toward −Y, and the policy's own heading correction turns the path toward +Y. That is why the useful strength is negative. Rotating only the observation by −9.2°, so the perceived heading turns toward −Y, reproduces the shift (+0.90 m/s, 20/20 pairs) with no failures.

Reproduce it with uv run --no-sync python scripts/audit_ant005_independent.py.

The scientific-rigor audit also confirms the effect under a randomization test and a displacement estimand: +17.4 m of extra world-Y travel. It finds that every HalfCheetah primary "speed effect" is a sign-independent slowdown.

Acceptance caveat: the recorded outcome remains confirmation_failed. Both arms have 4/30 failures, on partly different seeds; equal counts do not establish safety on every episode. A shared episode ends at step 34, before steering starts at step 100, violating the additional strict pre-onset criterion. Mean squared action increases 17.84%, and the original environment reward decreases. No formal replication or practical-target application has been completed. The experiment findings retain all results and controls; the paper / PDF covers all five experiments, including failures.

Baseline is on the left and steering is on the right. All three movies show the same predetermined validation seed 510000, replaying the saved trajectories exactly, as checked by the media and physics audit. They illustrate the effect; they are not fresh confirmation or application evidence.

Fixed, far-away camera: both panels share a stationary world frame and scale. The 20 m ground grid, position rings, and past-path trails are visual annotations; the robot's geometry and physics are unchanged. Camera configuration and replay checks record the measured extent and rendering setup.

Top-down tracking camera: see the gait and direction from above.

Original tracking camera: inspect the actual locomotion more closely.

Download the steering vector on Hugging Face, together with its manifest, scaling diagnostics, and locked selection. The published vector was downloaded back and its SHA256 matched the local artifact exactly. Video and vector reproduction instructions include pinned downloads and commands.

Install uv and Git, clone this repository, and run these commands from its root. The project uses Python 3.12 and a local .venv; uv resolves the interpreter and installs the dependencies specified by uv.lock. Both pinned policies and the Baukit intervention have been exercised on native Windows with CPU inference, and an Ant rollout video has been checked. Other operating systems require their own runtime verification.

git clone https://github.com/BurnyCoder/reinforcement-learning-locomotion-steering-vectors.git
cd reinforcement-learning-locomotion-steering-vectors
uv sync --locked
Copy-Item .env.example .env
uv run locomotion-steering run --model halfcheetah --run-dir runs/halfcheetah-classic-001
uv run locomotion-steering report --run-dir runs/halfcheetah-classic-001

The first run downloads the pinned public checkpoint, so internet access is required. Local experiment settings and episode counts are in .env; .env.example is the configuration reference. Current defaults include four rollout workers and a fitting speed floor of half the healthy fitting-window median. Rollout workers always use one Torch thread; STEERING_TORCH_THREADS sets the main process's threads, including replay and demonstrate. STEERING_SEED_OFFSET reserves fresh partitions for a new hypothesis. Keep secrets out of both tracked configuration and logs. A run performs actual simulator rollouts, including evaluation of available random and shuffled-label controls, so it takes longer than a single policy demonstration.

The run command writes numerical evidence under the supplied run directory. The separate report command renders report.md, report.pdf, and validation strength curves from that evidence. A completed run can legitimately report no eligible vectors or a failed confirmation. These outcomes are research findings, not useful steering successes.

uv run locomotion-steering run --model ant --run-dir runs/ant-classic-001
uv run locomotion-steering replay --run-dir runs/halfcheetah-classic-001 --vector speed --alpha 0.1
uv run locomotion-steering replay --run-dir runs/halfcheetah-classic-001 --vector speed --alpha 0.1 --off-at 700
uv run locomotion-steering report --run-dir runs/halfcheetah-classic-001
uv run pytest

Replay requires the named activation vector to have been extracted; inspect vectors.npz and vector_diagnostics.json or the report before choosing one. The example strength 0.1 is a demonstration argument, not a recommendation or a claim that it improves locomotion. Replay writes an MP4 and matching numerical NPZ under videos/; --off-at removes steering later in the episode, and --seed chooses the reset (default 40000). For a run with seed offset 0, seeds 40000–40009 are the cohort demonstrate reserves for application, so pick a seed outside every reserved cohort when the run may still be applied. Action-bias comparators are evaluated by the research pipeline and are not activation-vector replay inputs.

Re-run the same run command to resume completed episode artifacts under the saved model, configuration, and seed splits. Use a new run directory when changing the protocol. --stop-after diagnose, extract, calibrate, or confirm provides an inspection boundary. Report regeneration reads saved evidence without changing selection or running new evaluations.

To test an existing direction against a different behavioral outcome, use retarget with a separate run directory and a fresh STEERING_SEED_OFFSET in .env. For example, experiment 003 uses offset 200000 and the following command:

uv run locomotion-steering retarget --source-run runs/hc-running-002 --run-dir runs/hc-height-speed-003 --vector height --behavior speed

This reuses the exact source vector, its complete random/shuffled control family, and source fitting evidence. It recalibrates the new outcome on fresh episodes, preserving the source mapping in retarget.json. In the new bundle, vector keys use the target name: speed here identifies the imported height direction. Keep the source run's fitting trajectories available because shortlisted candidates require them for action-bias comparison. The experiment register links the actual outcomes and the uploaded 002 evidence.

After a speed or lateral candidate has passed confirmation and replication, uv run locomotion-steering demonstrate --run-dir runs/your-replicated-run tests its fixed, validation-derived target on ten new episodes. It restores the saved configuration, records numerical targeting and physical-quality checks, and renders the first three paired videos plus an off/on/off example. The command rejects runs without a supported replicated candidate. No completed experiment through 005 qualifies yet; this is an implemented workflow, not an application result. Exact acceptance rules are in methodology.

flowchart TD
    CLI[CLI and .env configuration] --> PLAN[Manifest and disjoint seed splits]
    PLAN --> LOAD[Pinned pretrained RL checkpoint]
    LOAD --> BASE[Unsteered diagnostic and fitting episodes]
    BASE --> FIT[Eligible behavior windows and contrastive mean vectors]
    FIT --> CONTROLS[Norm-matched random and shuffled-label vectors]
    CONTROLS --> VALIDATE[Paired validation strength sweep]
    VALIDATE --> BIAS[Fit and calibrate constant action-bias comparators]
    BIAS --> LOCK[Saved selection before confirmation]
    LOCK --> CONFIRM[Fresh paired confirmation episodes]
    CONFIRM --> REPLICATE[Fresh episode replication]
    REPLICATE --> RESULTS[Saved numerical results]
    RESULTS --> REPORT[Explicit report command: plots, Markdown and PDF]
    SOURCE[Saved family and source fitting evidence] --> RETARGET[Retarget: new outcome and fresh seed partitions]
    RETARGET --> VALIDATE
    REPLICATE --> DEMO[Demonstrate: fixed velocity target on fresh episodes]
    DEMO --> MEDIA[Application measurements and paired videos]
    BASE --> STORE[Compressed episode artifacts and timestamped logs]
    VALIDATE --> STORE
    CONFIRM --> STORE
    REPLICATE --> STORE

The thin pipeline coordinates reusable configuration, checkpoint/rollout, analysis, storage, and reporting modules. The first actor ReLU receives the intervention through the existing Baukit tracing utility. Ordinary Stable-Baselines3 prediction continues to handle action inference. The intervention is removed after use, and tests check that the original policy function is restored.

Preserve the run directory together with the exact Git commit and lockfile when sharing an experiment. New manifests automatically record the Git revision and normalized SHA256 hashes of owned source and dependency files; each resumed invocation logs its current code identity. Experiments 001–005 predate this field: their code_commit values were written into the manifests after creation, as each bundle README notes. These fingerprints exclude .env. Compare the saved checkpoint hash and configuration before interpreting a rerun. Episode replication reuses the same frozen checkpoint with fresh resets; it does not establish generalization across independently trained models.

The project currently implements classic activation-difference extraction on pretrained policies. Command-conditioned RL training, gradient-based extraction, and additional environments are research extensions to implement when diagnostics justify them. Local experimental compute has no overall time limit; each experiment still has a declared protocol and review point. One training seed per attempted policy is permitted.

Sources and reuse decisions distinguish upstream library behavior from this project's experimental adaptations. AGENTS.md contains contributor guidance. The experiment register defines terms and links individual run reports, including negative outcomes. The documentation fact-check and claim ledger connects substantive claims to artifacts, pinned sources, calculation procedures, corrections, and remaining provenance limits. It preserves the original outcomes and all recorded test attempts. The scientific-rigor audit checks whether the interpretations follow from the design. It covers estimator robustness, control specificity and alignment, and sign-dependent versus sign-independent responses. Run it with uv run --no-sync python scripts/audit_scientific_rigor.py. The independent fact-check recomputed the documented numbers from raw episodes and lists every correction; reproduce it with uv run --no-sync python scripts/factcheck_2026_10_22.py. The independent Ant audit adds three reusable modules:

• perception: rotates the Ant observation about the vertical axis, checked against rotated MuJoCo physics.
• specificity: rank tests, odd/even parts and SciPy bootstrap/sign-flip statistics.
• kinematics: travel angle, torso yaw and gait-axis velocity.