Skip to content

test(harness): run one conformance suite on every sandbox driver - #77

Merged
sanchitrk merged 2 commits into
canaryfrom
claude/cod-111-completion-8246b3
Oct 14, 2026
Merged

sanchitrk merged 2 commits into
canaryfrom
claude/cod-111-completion-8246b3

Conversation

@sanchitrk

@sanchitrk sanchitrk commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • New shared conformanceSpec (test/fixtures/conformance.spec.ts), run once per driver: host, memory, sqldb in test/sandbox.conformance.test.ts, and Vercel, Daytona from their E2E files on the sandbox each one already provisions. memory and sqldb are just-bash over a VFS, so they also cover just-bash.
  • Cases:
    • fs: read/write round-trip, overwrite, parent creation, read errors, buffers and byte ranges, scanLines corpus, stat/lstat incl. symlinks, readdir, exists, mkdir, rm
    • paths: relative → cwd, .. normalised, .. clamped at /
    • fs ↔ shell: each sees the other's writes
    • shell: exit codes and streams, per-command env, cwd, compound commands, pipelines and find, large output through ToolShell, typed timeout, pre-interrupted commands never start, mid-flight interrupt
    • host only: a child that ignores SIGTERM is killed
  • Each driver writes test/__artifacts__/sandbox.conformance.<driver>.json. Driver differences are declared options, never skipped: combinesOutput (Daytona), cancels (Daytona: false), inheritsHostEnv (host).
  • Replaces the old filesystem spec, the Flue parity matrix and the cancellation spec. remote.spec.ts keeps reattach/persist, GitHub push, and a check for the provider's baked env. withService moves to fixtures/service.ts.
  • The remote lifecycle E2E now calls wake and checks that create and wake each record codework.sandbox.provision.duration once.

Daytona fixes found by the suite

  • A non-recursive mkdir created missing parents, because the toolbox's createFolder is mkdir -p. It now runs mkdir in the shell.
  • rm of a missing path was not reported as not found. It now returns ENOENT.

Test plan

  • vp check, vp run typecheck
  • vp test test/sandbox. — 142 passed
  • Live conformance, reattach and push on Vercel and Daytona: 42 passed
  • sandbox.remote.driver.e2e.test.ts (provision metric): passed
  • Live bashPluginSpec and runner cycle on Vercel and Daytona: 13 passed

🤖 Generated with Claude Code

sanchitrk and others added 2 commits October 14, 2026 14:37
One shared spec covers files, paths, shell, timeouts and interrupts on host,
memory, sqldb, Vercel and Daytona, and writes a per-driver artifact. It
replaces the filesystem spec, the Flue parity matrix and the cancellation
spec, and moves the file and shell cases out of the remote spec. Drivers that
behave differently declare it as an option instead of skipping the case.

The suite found two Daytona bugs, fixed here: non-recursive mkdir created
missing parents, and rm of a missing path was not reported as not found.

The remote lifecycle test now checks that create and wake record
codework.sandbox.provision.duration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sanchitrk
sanchitrk merged commit f08d05b into canary Oct 14, 2026
2 checks passed
@sanchitrk
sanchitrk deleted the claude/cod-111-completion-8246b3 branch October 14, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant