Skip to content

Add fail-closed declarative NVX runtime support #66990

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lpcox merged 6 commits into from
Oct 8, 2026
Merged

Add fail-closed declarative NVX runtime support #66990

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a7c3e6f
Initial plan
Copilot Oct 8, 2026
510e919
Add fail-closed NVX runtime support
Copilot Oct 8, 2026
554c3d8
Potential fix for pull request finding
lpcox Oct 8, 2026
5db347b
Potential fix for pull request finding
lpcox Oct 8, 2026
5bee5b6
Potential fix for pull request finding
lpcox Oct 8, 2026
aa787b6
Make NVX configuration default-secure
Copilot Oct 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
254 changes: 254 additions & 0 deletions actions/setup/sh/nvx_host_preflight.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
#!/usr/bin/env bash
set -euo pipefail

echo "::group::NVX host and trusted artifact preflight"

trusted_dirs=(/usr/sbin /usr/bin /sbin /bin)
required_tools=(
bwrap flock getfacl getent gh groupdel id ip iptables mkfs.erofs mke2fs nft
setfacl setpriv sysctl useradd userdel
chmod install jq mktemp realpath sha256sum stat sudo uname
)
declare -A trusted_tools

tool_is_trusted_path() {
local candidate="$1"
for directory in "${trusted_dirs[@]}"; do
[[ "$candidate" == "$directory"/* ]] && return 0
done
return 1
}

for name in "${required_tools[@]}"; do
resolved=""
for directory in "${trusted_dirs[@]}"; do
candidate="${directory}/${name}"
if [[ -x "$candidate" ]]; then
resolved="$(/usr/bin/realpath -- "$candidate" 2>/dev/null || true)"
if [[ -n "$resolved" ]] && tool_is_trusted_path "$resolved"; then
break
fi
resolved=""
fi
done
if [[ -z "$resolved" || ! -f "$resolved" ]]; then
echo "::error::required trusted NVX host tool is missing or resolves outside /usr/sbin, /usr/bin, /sbin, or /bin: ${name}"
echo "Install the NVX Ubuntu host tools before the agent setup steps (acl bubblewrap e2fsprogs erofs-utils jq nftables uidmap)."
exit 1
fi
tool_owner="$(/usr/bin/stat -c '%u' -- "$resolved")"
tool_mode="$(/usr/bin/stat -c '%a' -- "$resolved")"
if [[ "$tool_owner" != "0" ]] || (( (8#$tool_mode & 0022) != 0 )) || (( (8#$tool_mode & 0111) == 0 )); then
echo "::error::required NVX host tool is not root-owned and non-writable by group/other: ${name} (${resolved})"
exit 1
fi
trusted_tools["$name"]="$resolved"
done

tool() {
local name="$1"
shift
"${trusted_tools[$name]}" "$@"
}

if [[ "$(tool uname -s)" != "Linux" ]]; then
echo "::error::NVX requires a Linux host."
exit 1
fi
if [[ "$(tool uname -m)" != "x86_64" ]]; then
echo "::error::NVX supports only Linux x86_64 hosts."
exit 1
fi
if [[ ! -c /dev/kvm || ! -r /dev/kvm || ! -w /dev/kvm ]]; then
echo "::error::NVX requires read/write access to the /dev/kvm character device."
exit 1
fi
if [[ ! -r /sys/fs/cgroup/cgroup.controllers ]]; then
echo "::error::NVX requires a readable cgroup v2 hierarchy."
exit 1
fi
cgroup_controllers="$(</sys/fs/cgroup/cgroup.controllers)"
for controller in cpu memory pids; do
if [[ " ${cgroup_controllers} " != *" ${controller} "* ]]; then
echo "::error::NVX requires the cgroup v2 ${controller} controller."
exit 1
fi
done
if [[ ! -r /proc/sys/kernel/seccomp/actions_avail ]] ||
[[ " $(</proc/sys/kernel/seccomp/actions_avail) " != *" kill_process "* ]]; then
echo "::error::NVX requires seccomp kill_process support."
exit 1
fi
if [[ "$(tool sudo -n "${trusted_tools[id]}" -u)" != "0" ]]; then
echo "::error::NVX requires non-interactive sudo so AWF can run with effective UID 0."
exit 1
fi

for variable in \
GH_AW_AWF_VERSION \
GH_AW_NVX_LAYER_SOURCE \
GH_AW_NVX_OPENVMM_SOURCE \
GH_AW_NVX_KERNEL_SOURCE \
GH_AW_NVX_INITRAMFS_SOURCE \
GH_AW_NVX_ARTIFACT_MANIFEST_SOURCE \
GH_AW_NVX_ARTIFACT_MANIFEST_BUNDLE_SOURCE \
GH_AW_NVX_SIGNER_WORKFLOW \
GH_AW_NVX_MOUNT_POLICY; do
if [[ -z "${!variable:-}" || "${!variable}" == *$'\n'* || "${!variable}" == *$'\r'* ]]; then
echo "::error::NVX preflight requires a non-empty, single-line ${variable} value."
exit 1
fi
done

version="${GH_AW_AWF_VERSION}"
if [[ "${version,,}" == "latest" ]]; then
echo "::error::NVX requires a concrete AWF release version so the attested manifest can be bound to that exact release."
exit 1
fi
[[ "$version" == v* ]] || version="v${version}"

if [[ "${GH_AW_NVX_MOUNT_POLICY}" == "workspace-and-tool-cache" ]]; then
tool_cache="${RUNNER_TOOL_CACHE:-${AGENT_TOOLSDIRECTORY:-}}"
if [[ -z "$tool_cache" || ! -d "$tool_cache" ]]; then
echo "::error::NVX mount-policy workspace-and-tool-cache requires RUNNER_TOOL_CACHE or AGENT_TOOLSDIRECTORY to exist."
exit 1
fi
fi

if [[ ! -d "$GH_AW_NVX_LAYER_SOURCE" || -L "$GH_AW_NVX_LAYER_SOURCE" ]]; then
echo "::error::NVX guest layer must be an existing non-symlink directory: ${GH_AW_NVX_LAYER_SOURCE}"
exit 1
fi
layer_path="$(tool realpath -- "$GH_AW_NVX_LAYER_SOURCE")"
if [[ "$layer_path" != /* ]]; then
echo "::error::NVX guest layer path must resolve to an absolute path."
exit 1
fi

artifact_paths=(
"$GH_AW_NVX_OPENVMM_SOURCE"
"$GH_AW_NVX_KERNEL_SOURCE"
"$GH_AW_NVX_INITRAMFS_SOURCE"
"$GH_AW_NVX_ARTIFACT_MANIFEST_SOURCE"
"$GH_AW_NVX_ARTIFACT_MANIFEST_BUNDLE_SOURCE"
)
artifact_names=(openvmm vmlinux initramfs.cpio.gz nvx-test-x86_64.manifest.json nvx-test-x86_64.manifest.sigstore.jsonl)
for index in "${!artifact_paths[@]}"; do
artifact="${artifact_paths[$index]}"
if [[ "$artifact" != /* || ! -f "$artifact" || -L "$artifact" || ! -s "$artifact" ]]; then
echo "::error::NVX trusted artifact must be an absolute path to a non-empty regular non-symlink file: ${artifact}"
exit 1
fi
if [[ "${artifact##*/}" != "${artifact_names[$index]}" ]]; then
echo "::error::NVX artifact ${index} must be named ${artifact_names[$index]}."
exit 1
fi
done

if ! "${trusted_tools[gh]}" attestation verify "$GH_AW_NVX_ARTIFACT_MANIFEST_SOURCE" \
--repo github/gh-aw-firewall \
--bundle "$GH_AW_NVX_ARTIFACT_MANIFEST_BUNDLE_SOURCE" \
--signer-workflow "$GH_AW_NVX_SIGNER_WORKFLOW" \
--deny-self-hosted-runners; then
Comment on lines +171 to +175
echo "::error::NVX manifest offline attestation verification failed; no unsigned or hash-only fallback is permitted."
exit 1
fi
Comment thread
Copilot marked this conversation as resolved.
Resolved

# shellcheck disable=SC2016
if ! "${trusted_tools[jq]}" -e \
--arg expectedTag "$version" \
--arg signer "$GH_AW_NVX_SIGNER_WORKFLOW" '
(keys | sort) == ["architecture", "artifacts", "release", "schemaVersion", "upstream"]
and .schemaVersion == 2
and .architecture == "x86_64"
and ((.release | keys | sort) == ["repository", "sourceCommit", "tag", "workflow"])
and .release.repository == "github/gh-aw-firewall"
and .release.workflow == $signer
and .release.tag == $expectedTag
and (.release.sourceCommit | type == "string" and test("^[a-f0-9]{40}$"))
and ((.upstream | keys | sort) == ["nvxCommit", "openvmmCommit", "releaseTag"])
and .upstream.releaseTag == "v0.1.0-dev.be859aa77ffa"
and .upstream.nvxCommit == "be859aa77ffa7a20f9ef50f68c5386acdfca9955"
and .upstream.openvmmCommit == "762bc1c7a203b16aee752324d6a4ab0bde1a713a"
and ((.artifacts | keys | sort) == ["initramfs", "kernel", "openvmm"])
and ([.artifacts[] | (keys | sort) == ["file", "sha256", "sizeBytes"]] | all)
and .artifacts.openvmm.file == "openvmm"
and .artifacts.kernel.file == "vmlinux"
and .artifacts.initramfs.file == "initramfs.cpio.gz"
and ([.artifacts[] | (.sizeBytes | type == "number" and . > 0 and floor == .)] | all)
and .artifacts.openvmm.sizeBytes <= 536870912
and .artifacts.kernel.sizeBytes <= 536870912
and .artifacts.initramfs.sizeBytes <= 1073741824
and ([.artifacts[] | (.sha256 | type == "string" and test("^[a-f0-9]{64}$"))] | all)
' "$GH_AW_NVX_ARTIFACT_MANIFEST_SOURCE" >/dev/null; then
echo "::error::NVX artifact manifest does not match the trusted AWF release and pinned upstream artifact contract."
exit 1
fi

for name in openvmm kernel initramfs; do
case "$name" in
openvmm) source="$GH_AW_NVX_OPENVMM_SOURCE"; role="openvmm" ;;
kernel) source="$GH_AW_NVX_KERNEL_SOURCE"; role="kernel" ;;
initramfs) source="$GH_AW_NVX_INITRAMFS_SOURCE"; role="initramfs" ;;
esac
expected_size="$("${trusted_tools[jq]}" -er ".artifacts.${role}.sizeBytes" "$GH_AW_NVX_ARTIFACT_MANIFEST_SOURCE")"
expected_digest="$("${trusted_tools[jq]}" -er ".artifacts.${role}.sha256" "$GH_AW_NVX_ARTIFACT_MANIFEST_SOURCE")"
Comment thread
Copilot marked this conversation as resolved.
Resolved
source_size="$(tool stat -c '%s' -- "$source")"
source_digest="$(tool sha256sum -- "$source")"
source_digest="${source_digest%% *}"
if [[ "$source_size" != "$expected_size" || "$source_digest" != "$expected_digest" ]]; then
echo "::error::NVX ${role} source artifact size or SHA-256 does not match the attested manifest."
exit 1
fi
done

stage_dir="$(tool sudo -n "${trusted_tools[mktemp]}" -d /tmp/gh-aw-nvx.XXXXXXXXXX)"
tool sudo -n "${trusted_tools[install]}" -o 0 -g 0 -m 0555 -- "$GH_AW_NVX_OPENVMM_SOURCE" "${stage_dir}/openvmm"
tool sudo -n "${trusted_tools[install]}" -o 0 -g 0 -m 0444 -- "$GH_AW_NVX_KERNEL_SOURCE" "${stage_dir}/vmlinux"
tool sudo -n "${trusted_tools[install]}" -o 0 -g 0 -m 0444 -- "$GH_AW_NVX_INITRAMFS_SOURCE" "${stage_dir}/initramfs.cpio.gz"
tool sudo -n "${trusted_tools[install]}" -o 0 -g 0 -m 0444 -- "$GH_AW_NVX_ARTIFACT_MANIFEST_SOURCE" "${stage_dir}/manifest.json"
tool sudo -n "${trusted_tools[install]}" -o 0 -g 0 -m 0444 -- "$GH_AW_NVX_ARTIFACT_MANIFEST_BUNDLE_SOURCE" "${stage_dir}/manifest.sigstore.jsonl"
tool sudo -n "${trusted_tools[chmod]}" 0555 "$stage_dir"

for name in openvmm kernel initramfs; do
case "$name" in
openvmm) file="${stage_dir}/openvmm"; role="openvmm" ;;
kernel) file="${stage_dir}/vmlinux"; role="kernel" ;;
initramfs) file="${stage_dir}/initramfs.cpio.gz"; role="initramfs" ;;
esac
expected_size="$("${trusted_tools[jq]}" -er ".artifacts.${role}.sizeBytes" "${stage_dir}/manifest.json")"
expected_digest="$("${trusted_tools[jq]}" -er ".artifacts.${role}.sha256" "${stage_dir}/manifest.json")"
Comment thread
Copilot marked this conversation as resolved.
Resolved
actual_size="$(tool stat -c '%s' -- "$file")"
actual_digest="$(tool sha256sum -- "$file")"
actual_digest="${actual_digest%% *}"
if [[ "$actual_size" != "$expected_size" || "$actual_digest" != "$expected_digest" ]]; then
echo "::error::NVX ${role} artifact size or SHA-256 does not match the attested manifest."
exit 1
fi
metadata="$(tool stat -c '%u:%a' -- "$file")"
expected_mode=444
[[ "$role" == openvmm ]] && expected_mode=555
if [[ "$metadata" != "0:${expected_mode}" ]]; then
echo "::error::NVX ${role} artifact is not root-owned with its required read-only mode."
exit 1
fi
done

manifest_metadata="$(tool stat -c '%u:%a' -- "${stage_dir}/manifest.json")"
bundle_metadata="$(tool stat -c '%u:%a' -- "${stage_dir}/manifest.sigstore.jsonl")"
if [[ "$manifest_metadata" != "0:444" || "$bundle_metadata" != "0:444" ]]; then
echo "::error::NVX manifest and attestation bundle must be root-owned read-only regular files."
exit 1
fi

{
printf 'GH_AW_NVX_LAYER=%s\n' "$layer_path"
printf 'GH_AW_NVX_OPENVMM=%s/openvmm\n' "$stage_dir"
printf 'GH_AW_NVX_KERNEL=%s/vmlinux\n' "$stage_dir"
printf 'GH_AW_NVX_INITRAMFS=%s/initramfs.cpio.gz\n' "$stage_dir"
printf 'GH_AW_NVX_ARTIFACT_MANIFEST=%s/manifest.json\n' "$stage_dir"
printf 'GH_AW_NVX_ARTIFACT_MANIFEST_BUNDLE=%s/manifest.sigstore.jsonl\n' "$stage_dir"
} >> "${GITHUB_ENV:?GITHUB_ENV is required}"

echo "NVX host and attested artifacts validated; AWF will fail closed if runtime startup fails."
echo "::endgroup::"
58 changes: 52 additions & 6 deletions docs/src/content/docs/reference/agent-runtimes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
---
title: Agent Runtime Selection
description: Choose and configure Docker, Cloud Hypervisor, or ARC DinD for an agentic workflow.
description: Choose and configure Docker, Cloud Hypervisor, NVX, or ARC DinD for an agentic workflow.
sidebar:
order: 1340
---

Agentic workflows use AWF (Agent Workflow Firewall) to isolate the agent. The supported agent runtime profiles are Docker, privileged Docker with legacy iptables networking, and preview Cloud Hypervisor. ARC DinD is a runner topology, not a value of `sandbox.agent.runtime`.
Agentic workflows use AWF (Agent Workflow Firewall) to isolate the agent. The supported agent runtime profiles are Docker, privileged Docker with legacy iptables networking, preview Cloud Hypervisor, and preview NVX. ARC DinD is a runner topology, not a value of `sandbox.agent.runtime`.

## Runtime and topology fields

| Field | Purpose | Supported values |
| --- | --- | --- |
| `sandbox.agent.runtime` | Selects the isolation backend for the main agent | `docker`, `docker-sudo-iptables`, `cloud-hypervisor`, or omitted for Docker |
| `sandbox.agent.runtime` | Selects the isolation backend for the main agent | `docker`, `docker-sudo-iptables`, `cloud-hypervisor`, `nvx`, or omitted for Docker |
| `runner.topology` | Describes how the runner reaches Docker | `arc-dind`, or omitted for a local Docker daemon |
| `runtimes` | Installs language toolchains such as Node.js, Python, and Go | Unrelated to agent isolation |

Expand All @@ -22,9 +22,10 @@ Agentic workflows use AWF (Agent Workflow Firewall) to isolate the agent. The su
| Docker | Linux namespaces, cgroups, and the host kernel | Linux and a usable Docker daemon | Broad compatibility; the agent shares the host kernel |
| Docker with legacy iptables | Docker with privileged AWF networking and host/service access | Linux, Docker, and host privileges | Supports host ports and Actions services but increases AWF privileges |
| Cloud Hypervisor (preview) | A KVM-backed microVM for the agent | GitHub-hosted Ubuntu x86_64 runner with `/dev/kvm` | Stronger isolation with strict host requirements |
| NVX (preview) | A one-shot NVX microVM with a workload-specific guest layer | Linux x86_64, effective root for AWF, read/write `/dev/kvm`, cgroup v2 (`cpu`, `memory`, `pids`), and seccomp `kill_process` | Strong isolation; requires trusted, attested AWF assets and explicit pre-agent provisioning |
| ARC DinD | Standard Docker agent container in a DinD sidecar | ARC or equivalent Kubernetes runner with a privileged DinD sidecar and shared work volume | Supports Kubernetes runner fleets but adds split-filesystem complexity |

Use `docker-sudo-iptables` only when the workflow requires `sandbox.agent.allow-host-ports` or access to published GitHub Actions `services:` ports. Use `cloud-hypervisor` only when the runner satisfies its preview requirements. Otherwise, omit `sandbox.agent.runtime` to use Docker.
Use `docker-sudo-iptables` only when the workflow requires `sandbox.agent.allow-host-ports` or access to published GitHub Actions `services:` ports. Use `cloud-hypervisor` or `nvx` only when the runner satisfies the selected preview runtime's requirements. Otherwise, omit `sandbox.agent.runtime` to use Docker.

## Requirements shared by all choices

Expand Down Expand Up @@ -103,6 +104,51 @@ Cloud Hypervisor is incompatible with:

If the runner does not meet these requirements, omit the runtime to use Docker.

## NVX (preview)

NVX runs the agent in AWF's fail-closed, one-shot microVM runtime. It requires AWF `v0.28.49` or newer and a Linux x86_64 host with effective UID 0 for AWF, read/write access to `/dev/kvm`, cgroup v2 `cpu`, `memory`, and `pids` controllers, and seccomp `kill_process` support. Generated preflight checks these requirements and fails the job with an actionable diagnostic; NVX never falls back to Docker or Cloud Hypervisor.

Configure the workload-specific guest layer and all five files from the exact AWF release before the generated AWF invocation. The artifact manifest and Sigstore bundle are verified offline with `gh attestation verify`; hashes alone are not accepted. The setup stages trusted files root-owned and read-only. Install these host packages before pre-agent setup: `acl bubblewrap e2fsprogs erofs-utils jq nftables uidmap`. The required host tools are resolved only from `/usr/sbin`, `/usr/bin`, `/sbin`, or `/bin`.

```aw wrap
---
on:
workflow_dispatch:
# A custom provisioning job must be listed in on.needs.
needs: [build_nvx_artifacts]
sandbox:
agent:
id: awf
runtime: nvx
nvx:
preview: true
network-isolation: true
api-proxy: true
mount-policy: workspace-only
layer-path: "${{ runner.temp }}/nvx/guest-layer"
openvmm-path: "${{ runner.temp }}/nvx/openvmm"
kernel-path: "${{ runner.temp }}/nvx/vmlinux"
initramfs-path: "${{ runner.temp }}/nvx/initramfs.cpio.gz"
artifact-manifest-path: "${{ runner.temp }}/nvx/nvx-test-x86_64.manifest.json"
artifact-manifest-bundle-path: "${{ runner.temp }}/nvx/nvx-test-x86_64.manifest.sigstore.jsonl"
signer-workflow: github/gh-aw-firewall/.github/workflows/release.yml
memory-mib: 512
memory-max-bytes: 536870912
pids-max: 128
# Optional:
# scratch-bytes: 536870912
# container-workdir: /workspace/build
---

Run the workload in the pre-provisioned NVX guest.
```

Path fields accept GitHub Actions expressions. Download and extract the trusted assets from the same pinned AWF release used by the workflow, then build the guest layer before the generated preflight step. `workspace-only` gives the guest the live workspace read-write at `/workspace`; `workspace-and-tool-cache` additionally exports the existing runner tool cache read-only.

For migration from the firewall repository's manual smoke workflows, keep each workflow's custom artifact-building job and workload-specific layer build, declare the artifact job under `on.needs`, and replace the manual AWF NVX command with the declarative runtime above. The Copilot smoke layer must still contain its pinned Copilot CLI; the build-test layer must still contain its pinned Node.js and Go toolchains. Download the workflow-attested AWF assets and restore the paths before pre-agent setup finishes. Do not pass provider credentials into the guest; the required API proxy keeps real credentials in its sidecar.

NVX rejects TTY/interactive execution, `runner.topology: arc-dind`, host or service port access, extra host mounts, Docker-in-Docker options, custom AWF arguments, DIFC proxy features, custom network subnets, DNS-over-HTTPS, and primary-agent enclaves. Do not disable preview, strict network isolation, or the API proxy. `workspace-and-tool-cache` fails if the runner tool cache is absent.

## ARC with Docker-in-Docker

Set `runner.topology: arc-dind` when the runner reaches Docker through a DinD sidecar:
Expand All @@ -117,7 +163,7 @@ runner:

The runner and DinD sidecar must share `/home/runner/_work`. `DOCKER_HOST` must point to the sidecar, and the sidecar must be privileged. The compiler stages binaries and configuration into daemon-visible paths and installs Docker Compose when needed.

Do not combine ARC DinD with `cloud-hypervisor`. Use the default Docker profile.
Do not combine ARC DinD with `cloud-hypervisor` or `nvx`. Use the default Docker profile.

## Troubleshooting

Expand All @@ -126,7 +172,7 @@ Debug runtime failures in this order:
1. Confirm the runner operating system and architecture.
2. Verify Docker access with `docker version`, `docker info`, and `docker compose version`.
3. Verify `DOCKER_HOST` and shared-volume visibility for ARC DinD.
4. Verify `/dev/kvm` and the GitHub-hosted runner requirement for Cloud Hypervisor.
4. Verify `/dev/kvm` and the host requirements for the selected microVM runtime.
5. Inspect the generated setup step that failed.
6. Recompile with the latest compatible gh-aw version.

Expand Down
3 changes: 3 additions & 0 deletions docs/src/content/docs/reference/sandbox.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,12 @@ strict: false
| `docker` (default) | Default Docker runtime, rootless AWF, network isolation |
| `docker-sudo-iptables` | Docker with privileged AWF, legacy `iptables` networking, and host/service access |
| `cloud-hypervisor` | Preview KVM runtime with its required privileged launcher |
| `nvx` | Fail-closed preview NVX one-shot microVM; requires explicit configuration and a Linux x86_64 KVM host |

Omitting `runtime` is equivalent to `runtime: docker`, which keeps the secure default.

The `nvx` runtime never falls back to another runtime. It requires a workload-specific guest layer, the AWF release's provenance bundle and manifest, and all three security opt-ins. The compiler checks the host and verifies the offline attestation before starting AWF. See [Agent Runtime Selection](/gh-aw/reference/agent-runtimes/#nvx-preview) for the complete configuration and compatibility requirements.

```yaml wrap
sandbox:
agent:
Expand Down
Loading
Loading