Skip to content

fix(ci): retry Nix shell and app dependency preparation - #4066

Merged
elezar merged 2 commits into
mainfrom
codex/fix-nix-shell-resume/mg
Oct 2, 2026
Merged

fix(ci): retry Nix shell and app dependency preparation#4066
elezar merged 2 commits into
mainfrom
codex/fix-nix-shell-resume/mg

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Oct 1, 2026 •
edited by elezar
Loading

Copy link
Copy Markdown
Contributor

Summary

Recover from intermittent Nix cache download failures during CI dependency preparation. A transport error after all payload bytes arrive can cause Nix to resume at EOF, receive HTTP 416, and fail before a check runs. Preparation gets one fresh invocation; checks and apps execute once.

Related Issue

No separate OpenShell issue required: localized CI reliability fix. Related upstream symptom report: NixOS/infra#1106.

Changes

The change is split into two commits:

  1. Move development-shell preparation into setup-nix, with opt-in prepare-shell and a shell-installable selector. Enable it for every shell-based workflow, including Cargo Deny, workflow security, Trivy, and VM-driver builds. Remove duplicate preparation from setup-rust.
  2. Retry the VM-runtime build once and prepare dependencies before each of the five nix run sites: integration tests, three artifact-generation apps, and protobuf compatibility. Expose existing artifact and protobuf app derivations as matching package outputs so preparation uses nix build --no-link before executing each app once.

Both preparation attempts remain in the normal log. A second failure fails the step. Any preparation failure is retried once; runtime failures from Cargo, lint, tests, artifact generation, and compatibility checks are not retried. Downloads initiated inside an app are outside this preparation retry.

Update CI documentation and the workflow-monitoring skill to reflect this behavior.

Testing

  • mise run pre-commit passes for both commits, including installed commit-hook checks.
  • YAML parsing and Bash syntax checks pass.
  • Changed workflows pass Actionlint; full-repository Actionlint reports existing unsupported concurrency.queue keys in publish-docs-website.yml and sync-docs.yml.
  • Mock Nix checks cover immediate shell success, second-attempt success, and two failures.
  • All five app sites pass 20 mock scenarios covering preparation success, retry recovery, persistent preparation failure, and runtime failure; apps execute at most once.
  • Nix evaluation confirms matching app/package paths for all four new package outputs on x86_64-linux, aarch64-linux, and aarch64-darwin.
  • Fresh GitHub CI and actual HTTP 416 recovery on these replacement commits have not yet been exercised.

Earlier log evidence motivating the workaround (these jobs tested the previous implementation):

  • macOS Prover: git documentation download resumed at offset 2556529, matching the compressed file size; 416 followed. Fresh command invocation recovered.
  • macOS Gateway: Perl download resumed at offset 12352024, matching the compressed file size; 416 followed. Fresh command invocation recovered.
  • Workflow Security / Actionlint: gettext download resumed at offset 4882844, matching the compressed file size; 416 followed. This previously uncovered entry point is now prepared through setup-nix.

Checklist

  • Conventional Commits with DCO sign-off.
  • CI documentation and contributor guidance updated.
  • Runtime commands are not retried.
  • Keep the PR draft pending fresh CI validation.

@copy-pr-bot

copy-pr-bot Bot commented Oct 1, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

🌿 Preview your docs: https://nvidia-preview-pr-4066.docs.buildwithfern.com/openshell

@matthewgrossman matthewgrossman added test:e2e Requires end-to-end coverage test:e2e-gpu Requires GPU end-to-end coverage labels Oct 1, 2026
@matthewgrossman

matthewgrossman commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 6c4bb76

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

Label test:e2e-gpu applied for 6c4bb76. Open Branch E2E Checks, find the run for commit 6c4bb76, and click Re-run all jobs to execute with the label set. The run will execute GPU E2E after building the required supervisor image once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

Label test:e2e applied for 6c4bb76. Open Branch E2E Checks, find the run for commit 6c4bb76, and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

This was referenced Oct 1, 2026
Closed
Merged
@matthewgrossman
matthewgrossman force-pushed the codex/fix-nix-shell-resume/mg branch from 6c4bb76 to d1d6d68 Compare October 1, 2026 22:05
@matthewgrossman

matthewgrossman commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test d1d6d68

@matthewgrossman matthewgrossman changed the title fix(ci): recover Nix shell downloads rejected on resume fix(ci): retry Nix shell preparation once Oct 2, 2026
elezar added 2 commits October 2, 2026 10:06
@elezar
fix(ci): prepare Nix development shells in setup-nix
4f1095d 
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
fix(ci): retry Nix builds before executing apps once
b1eef1c 
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar force-pushed the codex/fix-nix-shell-resume/mg branch from 4813a1c to b1eef1c Compare October 2, 2026 08:13
@elezar elezar changed the title fix(ci): retry Nix shell preparation once fix(ci): retry Nix shell and app dependency preparation Oct 2, 2026
@elezar

elezar commented Oct 2, 2026

Copy link
Copy Markdown
Member

The workaround targets a Nix download-resumption edge case: a transfer can report an error after receiving the full payload, then resume at EOF and fail with HTTP 416.

Evidence from our CI

All three jobs installed Nix 2.35.2. Each failed on a different .nar.zst object from cache.nixos.org, with the logged retry offset exactly matching the compressed FileSize in the object's narinfo:

Job log Failed output Retry offset / FileSize (bytes) Cache metadata
Workflow Security / Actionlint gettext 1.0 4,882,844 narinfo
macOS Prover git 2.55.0 documentation 2,556,529 narinfo
macOS Gateway Perl 5.42.0 12,352,024 narinfo

For example, the gettext log records Failed sending data to the peer, followed by retrying from offset 4882844 … (attempt 1/5), then HTTP 416. Both macOS jobs recovered after a fresh Nix invocation; Actionlint had no outer retry and failed before the check ran. These logs cover the previous PR implementation, not the replacement commits.

Why Nix's built-in retries do not recover

In Nix 2.35.2, request setup passes the bytes already streamed directly to CURLOPT_RESUME_FROM_LARGE. Retry eligibility does not guard against reaching EOF. The subsequent 416 is classified as a terminal 4xx error in error handling. Increasing download-attempts / filetransfer-retry-attempts therefore does not recover this case.

A separate local investigation reproduced the edge case on Nix 2.34.6 using an HTTP/1.1 chunked response that sends all 90,112 payload bytes but omits the final chunk. Nix reports a partial transfer, sends Range: bytes=90112-, and fails on 416. Control cases with a complete response and a genuinely partial response succeed, with payload SHA-256 verification. This is a synthetic reproduction of the EOF recovery problem; its transport error differs from CI, and the fixture has not yet been run against 2.35.2.

The initial CI transport failure remains unconfirmed. Nearby requests fail in bursts with SSL_read/EOF errors, but we do not have evidence assigning the cause to the CDN, runner network, HTTP/2, or libcurl. NixOS/infra #1106 reports the same 416 symptom without an established EOF diagnosis. No upstream Nix issue or released fix specifically for this mechanism has been confirmed; a candidate fix is being investigated separately.

Scope of these two commits

  • 4f1095dee: prepare development shells through setup-nix, with one fresh invocation on failure, before shell-based checks run.
  • b1eef1cf1: retry the VM-runtime build once; before each nix run, build its package with one retry and then execute the app once.

Any preparation failure is retried once, and persistent failures still fail CI. Cargo, checks, tests, and artifact-generation runtime failures are not retried. Downloads initiated inside an app remain outside this preparation workaround. --fallback would instead permit source builds after a failed substitute; this workaround first gives cached dependencies another download attempt.

@elezar

elezar commented Oct 2, 2026

Copy link
Copy Markdown
Member

/ok-to-test b1eef1c

@elezar elezar mentioned this pull request Oct 2, 2026
Draft
@elezar
elezar marked this pull request as ready for review October 2, 2026 08:45
@elezar
elezar requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners October 2, 2026 08:45
@elezar
elezar added this pull request to the merge queue Oct 2, 2026
Merged via the queue into main with commit 6048bed Oct 2, 2026
115 checks passed
@elezar
elezar deleted the codex/fix-nix-shell-resume/mg branch October 2, 2026 11:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SDAChess SDAChess SDAChess approved these changes
@mrunalp mrunalp Awaiting requested review from mrunalp mrunalp is a code owner
@sjenning sjenning Awaiting requested review from sjenning sjenning is a code owner
@derekwaynecarr derekwaynecarr Awaiting requested review from derekwaynecarr derekwaynecarr is a code owner

Assignees

No one assigned

Labels

test:e2e Requires end-to-end coverage test:e2e-gpu Requires GPU end-to-end coverage

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@matthewgrossman @elezar @SDAChess