Skip to content

fix(benchmarks): select cached plugin versions numerically - #970

Open
rudycelekli wants to merge 1 commit into
DietrichGebert:mainfrom
rudycelekli:fix/agentic-cache-version-order-20261001
Open

rudycelekli wants to merge 1 commit into
DietrichGebert:mainfrom
rudycelekli:fix/agentic-cache-version-order-20261001

Conversation

@rudycelekli

Copy link
Copy Markdown

Summary

Resolve cached plugin releases by numeric version components so the agentic benchmark loads the latest installed release. Environment overrides still take precedence; cache selection and activation keep their existing contract.

Reproduction

With both ~/.claude/plugins/cache/ponytail/ponytail/4.9.0 and 4.10.0 installed, the existing lexical ordering chooses 4.9.0. I reproduced this through the public benchmarks/agentic/run.py runner with one cache task and an offline CLI fixture that records the actual --plugin-dir argument. The fixed runner chooses 4.10.0.

The regression check also covers patch versions 4.10.2 / 4.10.12 and preserves an explicit plugin directory override.

Scope and prior work

The numeric ordering idea also appeared in #903, which its author closed for a task-scope misunderstanding. This proposal isolates that still-reproducible bug, leaves checkout activation/provenance/rescore metadata out of scope, and does not overlap the active selftest environment restoration in #717 or unused ARM cleanup in #668.

Numeric chunks cover the current release directories; this deliberately does not implement full prerelease SemVer ordering. The code states when a version parser would be needed.

Verification

  • Regression check fails against the unchanged main implementation and passes after the fix.
  • Actual public runner red/green replay records the selected cache directory. No real model or provider was invoked; outbound network was denied.
  • npm test: 122 passed, zero failures/skips (96 root, 23 Pi, 3 MCP).
  • node scripts/check-rule-copies.js: passed.
  • node scripts/check-versions.js: passed; all eight version files remain 4.10.0.
  • git diff --check: passed.

AI-assisted contribution, independently reviewed and replayed before publication.

Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant