Skip to content

Latest commit

 

History

History
400 lines (337 loc) · 20.9 KB

File metadata and controls

400 lines (337 loc) · 20.9 KB

Windows WHPX

A3S Box runs Linux OCI workloads directly on x86_64 Windows through the Windows Hypervisor Platform (WHPX) backend in libkrun. WSL is not part of the runtime path.

Requirements

  • x86_64 Windows 10 or Windows 11;
  • hardware virtualization enabled in firmware;
  • a working WHPX-capable hypervisor stack (see below);
  • Windows Developer Mode enabled, or the A3S Box service identity granted SeCreateSymbolicLinkPrivilege (A3S Box temporarily enables an assigned but disabled privilege only while probing the capability or extracting an OCI layer);
  • the A3S Box Windows binaries and their matching runtime DLLs.

A3S Box talks to the Windows Hypervisor Platform APIs. Operators commonly enable that surface with:

Enable-WindowsOptionalFeature -Online -FeatureName HypervisorPlatform

Restart Windows if the feature manager requests it. On hosts that already run full Microsoft Hyper-V (Microsoft-Hyper-V / Microsoft-Hyper-V-Hypervisor Enabled), WHPX workloads can succeed even when the standalone HypervisorPlatform optional feature still reports Disabled. Treat a3s-box info as the authoritative preflight: it must report Virtualization: WHPX (or VM backend: whpx) and OCI symlink support: available before starting workloads. Do not fail a host solely because Get-WindowsOptionalFeature -FeatureName HypervisorPlatform returns Disabled when info already proves WHPX is available.

The WHPX probe uses the same scoped privilege implementation as layer extraction, so an assigned but initially disabled service-token privilege is reported accurately.

If the probe reports that SeCreateSymbolicLinkPrivilege is enabled but link creation is still denied, check the A3S home ACL and endpoint-protection policy. Allow the approved a3s-box.exe binary and its A3S home directory according to the organization's security policy; do not flatten or replace OCI links.

Linux OCI images commonly contain symbolic links for paths such as /bin, dynamic loaders, and shared libraries. The Windows rootfs is backed by NTFS and served to the guest through virtio-fs, so A3S Box must preserve those entries as real symbolic links. Enable Developer Mode in Settings > System > Advanced > For developers (called Settings > System > For developers before Windows 11 25H2). Once enabled, A3S Box can run as a normal, non-elevated process. See Microsoft's Developer Mode instructions and CreateSymbolicLink documentation.

If Developer Mode is disabled and the process token lacks the privilege, or the target directory denies link creation, image extraction fails with an explicit ERROR_ACCESS_DENIED (5) / ERROR_PRIVILEGE_NOT_HELD (1314) diagnostic. This is intentional: replacing a symbolic link with a copied file, hard link, or directory junction would change OCI layer, whiteout, and guest path-resolution semantics.

Runtime package

A runnable Windows package contains:

a3s-box.exe
a3s-box-shim.exe
a3s-box-guest-init
krun.dll
libkrunfw.dll

a3s-box-guest-init is a static Linux executable used as PID 1 inside the MicroVM. krun.dll provides the native WHPX VMM, while libkrunfw.dll carries the Linux guest kernel. Both DLLs must remain next to the Windows executables so the Windows loader can resolve them. Zig cross-links the Linux guest binaries; it does not compile the Linux kernel.

For a source build:

winget install --id zig.zig --exact --version 0.16.0
cd src
powershell.exe -NoProfile -ExecutionPolicy Bypass -File deps/libkrun-sys/vendor/libkrun/scripts/build-windows-init.ps1
cargo install cargo-zigbuild
cargo zigbuild --release -p a3s-box-guest-init --target x86_64-unknown-linux-musl
cargo build --release -p a3s-box-cli -p a3s-box-shim

The nested script is required before rebuilding krun.dll from a fresh submodule checkout. It produces the ignored, stripped init/init payload used by libkrun's Windows wrapper; release workflows never substitute an empty file.

The build copies krun.dll and libkrunfw.dll next to the native binaries in the Cargo target directory.

Current support boundary

Capability Windows status
Pull and run Linux OCI images Implemented; Alpine 3.20 validated on real WHPX
Foreground and detached run Validated
ps, logs, inspect, and read-only attach Validated, including separated stdout/stderr and structured JSON logs
Non-interactive post-boot exec Validated through the shim-owned local named pipe and guest control tunnel, including environment propagation and repeated commands
cp, top, and stats --no-stream Validated for bidirectional single-file copy, guest process listing, and guest PID accounting
Workload exit codes Validated for foreground and detached workloads
Long workload arguments Validated with a 4,096-byte argument staged outside the bounded guest kernel command line
Graceful stop and cleanup Validated through the guest control channel with configured signal delivery, bounded force termination, and no residual shim or forwarding worker
vCPUs Exactly one; omitting --cpus selects the Windows default of 1
Published TCP ports Validated through the Windows named-pipe bridge, including sequential connections
Bind mounts Validated for drive-letter directory and single-file sources, including read-only enforcement
Named volumes Validated across stop and restart, including explicit removal
Initialization scripts Validated for a read-only host-provided script/config plus a named state volume, including exact success, exit 42 failure, and persisted evidence from both attempts
POSIX ownership and modes Validated for chmod, chown, and umask-created files and directories across clean stop, restart, and commit
diff, export, stopped-box commit, and stopped-box filesystem snapshots Validated through clean-stop metadata capture, committed-image re-run, snapshot restore/restart, guest-visible restored content (exec), and stopped re-export of a snapshot-restored box after a second boot generation (legacy unmanaged restore stop delivers stop.signal so guest terminal metadata is rewritten). Running-box host-path capture remains unavailable because WHPX has no post-boot guest archive channel.
Container health checks Not currently supported; --health-* requests and persisted health checks fail before workload start
Bridge networks and Compose service networking Not currently supported on Windows
Interactive PTY (run -t, attach -it, exec -it, and shell) Not currently supported on Windows; these commands fail before box creation or lifecycle mutation. Non-interactive exec is supported
Shared-kernel Sandbox isolation Not supported on Windows; --isolation sandbox fails before box creation
Memory snapshot-fork, TEE, and CRI Not supported on Windows; --tee / --tee-simulate fail before box creation
pause / unpause Not supported on Windows MicroVM/WHPX; commands fail closed before lifecycle mutation

Requests such as --cpus 2, --health-cmd ..., --isolation sandbox, --tee, or run -t fail before image pull / box creation. A missing WHPX hypervisor fails in that same preflight, before a box directory or VM is created, and the diagnostic does not redirect the operator to WSL. The default boot kernel is the one bundled in libkrunfw.dll; A3S_BOX_KERNEL is an explicit override and is not resolved from a WSL package path. --no-healthcheck remains available to disable an image-defined health check. Qualification-mode run and create also require the configured OCI service to advertise one launch-ready DedicatedVm driver before named-volume creation or image-cache access.

Smoke test

The following paths were validated on July 20–22 and again on August 1, 2026, on Windows build 26200 with an AMD Ryzen 7 9800X3D and a WHPX-capable hypervisor stack (full Hyper-V and/or the HypervisorPlatform optional feature):

# Success, stdout, and stderr
a3s-box run --name whpx-ok alpine:3.20 -- /bin/sh -c `
  'echo WHPX_OK; echo WHPX_STDERR_OK >&2; uname -srm'

# Real non-zero exit propagation
a3s-box run --name whpx-exit alpine:3.20 -- /bin/sh -c 'exit 7'
$LASTEXITCODE  # 7

# Detached result reconciliation
a3s-box run -d --name whpx-detached alpine:3.20 -- /bin/sh -c `
  'sleep 1; echo DETACHED_OK; exit 3'
a3s-box ps -a
a3s-box logs whpx-detached
a3s-box inspect whpx-detached

The validation booted the kernel bundled in libkrunfw.dll as Linux 6.12.91. The /sbin/init installed into the test rootfs matched the locally Zig-cross-linked guest-init byte for byte. Foreground exit 7 and detached exit 3 both reached the host unchanged. The validation also ran all 113 Linux guest-init unit tests inside a real WHPX guest, sent multiple requests through the same published TCP port, exercised Windows directory and single-file bind mounts, verified graceful and forced-stop cleanup without orphan processes, restarted a named volume, and restored and restarted a filesystem snapshot.

The August 1 qualification additionally exercised repeated post-boot exec, bidirectional single-file cp, top, and guest PID-aware stats --no-stream. All 12 selected tests passed in 563.541 seconds against the OCI archive with SHA-256 45a567b46d75167f02a0a0042781fed3dfa5835b2b4c7c85fa1b8259f67aa27d. Peak observed runtime usage was 170,004,480 bytes and 960 handles, and both the starting and final A3S process inventories were empty.

Standard Compose services create a bridge network by default. Because native WHPX bridge networking is not implemented, Compose workload startup remains outside the current Windows support boundary even when a Compose file contains only one service. compose up rejects this platform combination before image resolution, network creation, box-directory creation, or VM startup.

First-principles acceptance matrix

Release claims for Windows WHPX must cover both the supported positive subset and fail-closed negatives. Tracked as Box issues #252, #255, and #257, the minimum named rows are:

Family Examples
WIN-HOST-* info reports WHPX; package DLLs co-located; docs match a host that can actually run
WIN-POS-* foreground/detached run, exec, cp/top/stats, volumes, bind RO, ports, snapshot, commit, rename, wait/kill exit-code honesty
WIN-NEG-* sandbox, tee, health, bridge, compose, PTY/shell, pause/unpause, live container-update, warm pool
WIN-SLO-* no residual processes after cases; exec-ready / stop-delivery WARNs stay off the happy path

CLI coverage for several negatives lives in src/cli/tests/command_coverage.rs (test_windows_*). Real WHPX soak remains the evidence gate for positives (scripts/windows-whpx-soak.ps1).

Additional fail-closed negatives covered after #259–#263:

ID Assertion
WIN-NEG-BRIDGE-* --network bridge fails before pull; network create rejects bridge
WIN-NEG-POOL-01 pool start exits non-zero immediately
WIN-NEG-CUPD-01 container-update --cpus 2 rejects and does not persist

Tip: cargo test -p a3s-box-cli --test command_coverage test_windows_ on Box 79b25590… worktree → 4/4 pass (sandbox/TEE/bridge/PTY/pool/health/Compose/ live container-update). Tip: scripts/test-install.ps1 on tip → pass (install.ps1 tests passed for windows-x86_64; digest/SHA verify + managed install layout). Does not claim Enterprise GA.

WHPX soak validation

Run the Windows-specific soak harness from the Box repository root on an otherwise idle WHPX host. It builds the current guest-init and Windows binaries (unless -SkipBuild), then precompiles the real smoke executable and repeatedly exercises the supported lifecycle, logs, exit-code, long-argv, post-boot exec, bidirectional single-file copy, top, guest PID-aware stats, published-port, bind-mount (:ro via BindFlt), virtiofs tar stress (--virtiofs-cache=none), volume-backed init script, named-volume, commit, and filesystem snapshot paths.

Pass -SkipVirtiofsStress to omit the ~100s virtiofs tar case from short rehearsals. The default matrix includes it after Windows tip-prove.

This runner supplies the WIN-01 lane in the Cross-Capability Soak Test Plan. Its evidence proves only the documented Windows subset; unsupported Windows features require fail-closed functional tests rather than being inferred from another host.

Tip (Box c8c2c1dc…): one-iteration soak summary SHA-256 93c085108896c24620c506ead1f517a28a5b1114c2d31aeecc02353f4cc51f35 (result=pass, verification=pass, nine tests). Tip (Box 1c741962…): three-iteration longitudinal summary SHA-256 0ff23f04a470e6de12f616b3a9d29ea4aa5be10d7a90fa4b152351e313bae52e (27 tests, ~16 min). Tip (Box 07514c79…, BindFlt :ro + bind-mount): one-iteration soak summary SHA-256 a90e771533e4d17064481889811f22a2e01c499872078e651fd6b2955afce00f (result=pass, verification=pass, ten tests, ~5.5 min). Tip (volume-backed init on Windows): one-iteration soak summary SHA-256 6aea2e8553693a1a3790b83f41bc9ca617a57a0675695fc33be6a7548e5840d6 (result=pass, verification=pass, eleven tests, ~6 min). Tip (G2 7200s): summary SHA-256 49e0c3820401e89a522bcb4034b591e9644c2efd4a5bc2721d9169e8e9da5a96 (result=pass, verification=pass, 22×11=242 tests, ~123 min). Tip (virtiofs tar on Windows): one-iteration soak summary SHA-256 0452f1370ecdaf3cd4b4e2b72d3d032241ed173e36f6c83300a1329954e6ba95 (result=pass, verification=pass, twelve tests, ~7 min). Tip (R24 86400s): summary SHA-256 9b5e1f3c5a1ccf31496f5f253605f044b293f1d816ec95ff1372f1042da86a1d (result=pass, verification=pass, 188×12=2256 tests, ~24.0 h). Tip (Box c689f5dd…, writable-bind O_TRUNC in matrix): one-iteration soak summary SHA-256 e88f1ca3df0b63213a0933079273955aae294fa26e954112f318bcfe2ef6f955 (result=pass, verification=pass, thirteen tests, ~7.7 min). Tip (G2 7200s on 13-test O_TRUNC matrix, Box cfae3a02…, solo idle WHPX): summary SHA-256 6222d0b003126bce412f476318200a32cdd0baae0fb6c547fd566bbce0c962d8 (result=pass, verification=pass, 16×13=208 tests, ~126.8 min). R24 13-test O_TRUNC tip digest is not yet claimed: prior attempt win01-r24-13test-otrunc-retry-20260926T012300 failed mid-iter-18 after 17 clean iterations when Process.Start hit a core_smoke image sharing violation; the harness now retries that Start path. Does not claim Enterprise GA or close B3/B4/B5/B6.

.\scripts\windows-whpx-soak.ps1 `
  -ImageTar C:\images\alpine-3.20.tar `
  -Iterations 1

# Two-hour gate. The current iteration is allowed to finish after the deadline.
.\scripts\windows-whpx-soak.ps1 `
  -ImageTar C:\images\alpine-3.20.tar `
  -Iterations 0 `
  -DurationSeconds 7200

Evidence is written under src/target/a3s-box-whpx-soak/ by default. Each test has its own log. summary.json, operations.tsv, resource-samples.tsv, start/final inventories, host.json, and verify.out record the commit, image digest, timings, peak runtime working set, handles, process counts, failures, and cleanup. The runner requires no active A3S Box or A3S OCI Runtime process at startup, verifies the same invariant after every test, and fails when requested/completed counts or resource guardrails drift.

The default matrix includes a 4,096-byte workload argument, POSIX ownership/mode replay through restart and commit, BindFlt :ro bind-mounts, writable-bind O_TRUNC overwrite, virtiofs tar stress, and volume-backed init (thirteen tests). Use -ListTests to inspect the exact selection.

Long-running gates need extra WHPX settle margin. The harness waits 8 seconds between tests (-InterTestDelayMilliseconds), and it sets A3S_EXEC_READY_TIMEOUT_MS=60000 for smoke children unless the caller already set it. With 3–5 seconds of delay and the 15-second product default for exec readiness, R24 runs failed after 1.5–2 hours: single-file :ro failed with exit code 101, and volume-backed init was force-killed before its exec server came up. The 15-second product default is unchanged.

The virtio-fs case intentionally scans 2,048 files five times with cache mode none. Real WHPX validation took 373 seconds on the host described above, so that test has an independent 900-second default budget. -SkipVirtiofsStress is suitable for a short functional rehearsal, not for release soak evidence.

Packaged WHPX Host install layout (production cutover)

Durably install the OCI windows-whpx-qualification artifact without flattening bin/ into the same directory as system-image/:

install-root/
  bin/
    a3s-box.exe                 # tip Box CLI (optional beside Host)
    a3s-oci.exe
    a3s-oci-krun-shim.exe
    krun.dll
    libkrunfw.dll
  system-image/
    system-image.json
    a3s-oci-system.ext4…
  bootstrap-vm-rootfs/          # empty seed; Box materializes under A3S home

Equivalent A3S-home layout: %USERPROFILE%\.a3s\bin\ for Host binaries and %USERPROFILE%\.a3s\share\a3s\system-image\ for the immutable image. Nesting system-image/ under the shim directory fails closed at discovery (and would fail later inside WindowsSystemImage::load).

Box-to-OCI Runtime product qualification

The unified-runtime vertical slice has a separate, build-free hardware gate. Download the Box windows-whpx artifact and the pinned OCI Runtime windows-whpx-qualification and guest-agents-musl artifacts without renaming or removing their artifact-manifest.json files. Use the fixed Alpine 3.22.5 x86_64 minirootfs archive, whose SHA-256 is 4b4daa9fe2fc696c4919c4412a4c3d3e770d8fb70292a004a2c72f5096175282.

.\scripts\windows-whpx-oci-qualification.ps1 `
  -BoxArtifactDirectory C:\artifacts\box-windows `
  -OciWindowsArtifactDirectory C:\artifacts\oci-windows `
  -OciGuestArtifactDirectory C:\artifacts\oci-agents `
  -RootfsArchive C:\images\alpine-minirootfs-3.22.5-x86_64.tar.gz

The runner verifies the checkout and pinned OCI source commits, requires the two OCI artifacts to share one workflow commit and run ID, and rehashes every input before staging it. Before starting the service, it runs a3s-box info from the staged artifact and requires OCI symlink support: available; the captured box-info.stdout.log distinguishes missing Windows capability from an ACL or endpoint-protection denial. It then starts the protected named-pipe service, imports the rootfs into a dedicated Box home without registry access, and runs the public Box manager through idempotent create, manager reopen and reconcile, WHPX start/wait, exact exit code 23, and generation-fenced delete replay. Success also requires no Box execution directory, OCI generation share, bundle handoff, or A3S host process to remain. report.json and summary.json use the versioned a3s.box.windows-whpx-oci-qualification.v1 and a3s.box.windows-whpx-oci-qualification-run.v1 schemas respectively. Preparation and start have a 30-minute bound. On Windows, handoff validation accepts the ordinary and \\?\ namespace spellings only when they name the same exact container/create-operation path; a different operation suffix or a filesystem alias remains invalid.

The first artifact-bound product gate passed on real x86_64 Windows/WHPX on August 4, 2026. It used Box 52a2cfe4ee6693c9cc3a88df1b922bc1825b2deb from CI run 30889251291 and pinned OCI Runtime 08c145d8ce5d06d5f28587226be822a2ab43b299 from main run 30881404238. The twelve-minute run observed the exact libkrun-whpx/dedicated-vm binding, running state, exit code 23, create and delete replay, manager-restart reconciliation, complete lifecycle-directory cleanup, and no residual A3S processes.

The exact post-merge Box main artifact aaf9e615ee8bb5e22a5214ca09d7e426701f2d58 from main CI run 30898682738 subsequently passed the same complete gate against the pinned OCI Runtime main artifacts. This verifies that the merge result, artifact manifest, and tested WHPX lifecycle all refer to published main revisions rather than only the PR head.

Diagnostics and kernel override

The default WHPX boot path automatically selects the current reliable single-vCPU, legacy-PIC kernel configuration. Do not set a custom kernel for normal operation.

For kernel debugging only, A3S_BOX_KERNEL can point to an x86_64 ELF vmlinux or a supported PE/COFF kernel image:

$env:A3S_BOX_KERNEL = 'C:\path\to\vmlinux'

LIBKRUN_WINDOWS_KERNEL_CMDLINE_APPEND is also an expert override for the additional Windows kernel command line. When it is absent, A3S Box supplies noapic; an explicitly set value, including an empty value, replaces that default.

If a boot fails, inspect the per-box files below $env:A3S_HOME\boxes\<id>\logs\ and rootfs\init-rust.log. A package missing either krun.dll or libkrunfw.dll is incomplete.