Repository navigation
test(aikit): Anthropic e2e model matrix, with the fixes it found - #42
Merged
Merged
Conversation
… handling - modelgen: live-verified thinking levels for gpt-5.4 through gpt-6 (no minimal; max only on gpt-5.6/gpt-6; none on Codex gpt-5.5/5.6-luna/6-luna) and add Codex gpt-6-luna - bump @ai-sdk/openai to 4.0.81 so gpt-6-luna accepts reasoning "none" - floor the context-clamped max tokens at 16 so an overflowing prompt surfaces as an overflow instead of an invalid max_output_tokens - normalize Codex call|item tool call IDs for OpenAI targets (64-char cap) - classify in-stream StreamProviderError like APICallError Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every OpenAI suite runs on gpt-6-luna through gpt-5.4-nano and every Codex suite on gpt-6-luna, gpt-5.6-luna and gpt-5.5. Adds thinking-level, context overflow, handoff and prompt-cache suites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ID handoff
- modelgen: live-verified Anthropic levels (max on 4.6; xhigh and max on
4.7/4.8, Sonnet/Opus 5 and Fable 5; 5.5 and Fable cannot disable
thinking) and Sonnet 4.5's 200k direct-API window
- skip `thinking: {type: "disabled"}` for Claude models that reject it
- normalize Codex call|item tool call IDs for Anthropic targets, leaving
already-valid IDs untouched
- run the Anthropic e2e suites across a model matrix and add Anthropic to
the thinking-level, overflow, handoff and cache suites
- document the OpenAI and Anthropic changes in the changelog
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Runs every Anthropic e2e suite against a model matrix and fixes the bugs those live runs surfaced. Stacked on #41 (OpenAI + Codex matrix), whose two commits are included here; the changelog covers both.
Matrix (
test/utils/llm.ts):claude-sonnet-4-6,claude-sonnet-5,claude-sonnet-5-5,claude-opus-4-6,claude-opus-4-8,claude-opus-5-5,claude-fable-5-1, then the 4.5 generationclaude-sonnet-4-5-20250929,claude-haiku-4-5-20251001.Anthropic requests send no reasoning by default (exercising the thinking-off path) and
highfor thinking and multi-turn. Sonnet 5, Sonnet 5.5 and Fable usemaxin thinking tests: on short prompts they skip thinking belowmax(confirmed against the raw API).Anthropic is added to the thinking-level, overflow, handoff and cache suites. Handoff covers model switches plus Anthropic → OpenAI, OpenAI → Anthropic and Codex → Anthropic; caching checks a write then a read for both
shortandlongretention.xhigh.e2e.test.tsis removed; the thinking-level suite covers it.Fixes
thinking: {type: "disabled"}, which they rejectdisabledProviderOptionsrespects it for Anthropicxhigh/maxnever offered on 4.7/4.8, Sonnet/Opus 5.x, Fable;maxmissing on 4.6modelgen.tsmodelgen.tscall|itemtool IDs fail Anthropic's^[a-zA-Z0-9_-]+$transform.ts; valid IDs pass through unchangedTest plan
vp check,vp run typecheckpnpm test(246 unit)vp test test/e2e -t "OpenAI|Anthropic"(705 live tests across OpenAI, Codex and Anthropic)Regenerate the gitignored catalog before running the e2e suites:
🤖 Generated with Claude Code