Repository navigation
test(aikit): OpenAI and Codex e2e model matrix, with the fixes it found - #41
Merged
Merged
Conversation
… handling - modelgen: live-verified thinking levels for gpt-5.4 through gpt-6 (no minimal; max only on gpt-5.6/gpt-6; none on Codex gpt-5.5/5.6-luna/6-luna) and add Codex gpt-6-luna - bump @ai-sdk/openai to 4.0.81 so gpt-6-luna accepts reasoning "none" - floor the context-clamped max tokens at 16 so an overflowing prompt surfaces as an overflow instead of an invalid max_output_tokens - normalize Codex call|item tool call IDs for OpenAI targets (64-char cap) - classify in-stream StreamProviderError like APICallError Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every OpenAI suite runs on gpt-6-luna through gpt-5.4-nano and every Codex suite on gpt-6-luna, gpt-5.6-luna and gpt-5.5. Adds thinking-level, context overflow, handoff and prompt-cache suites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Runs every OpenAI and Codex e2e suite against a model matrix, adds new live suites, and fixes the bugs those runs surfaced.
Matrices (
test/utils/llm.ts)gpt-6-luna,gpt-5.6-luna,gpt-5.5,gpt-5.4,gpt-5.4-mini,gpt-5.4-nanogpt-6-luna,gpt-5.6-luna,gpt-5.5. The Codex backend rejectsgpt-5.4,gpt-5.4-miniandgpt-5.4-nanofor ChatGPT accounts ("not supported when using Codex with a ChatGPT account").Reasoning is
lowby default,mediumfor thinking tests, andhighfor multi-turn, replay and handoff (smaller models skip reasoning on easy prompts belowhigh).New suites
thinking.levels: every model × every reasoning level plus off, liveoverflow: an oversized prompt must return an errorisContextOverflowrecognises; writes__artifacts__/overflow.*.jsonhandoff: reasoning-only aborted turn replay, mid-conversation model switch, Codex ↔ OpenAI in both directionscache: a follow-up request in the samesessionIdreads the cached prefixFixes
minimalon every matrix model;maxonly works on gpt-5.6/gpt-6;gpt-6-lunahad no level map; Codexgpt-6-lunamissing; Codex models couldn't turn reasoning off although the API acceptsnonemodelgen.tscatalog, verified against the live APIs@ai-sdk/openai4.0.66 droppedreasoningEffort: "none"forgpt-6-luna@ai-sdk/provider4.0.19 to align types)max_output_tokens: 1, rejected by OpenAI (min 16), hiding the overflowMIN_MAX_TOKENS1 → 16 inshared.tscall|itemtool-call IDs (83 chars) sent to OpenAI exceed its 64-charcall_idlimittransform.tsStreamProviderError), classified asUnknownAPICallErrorinfailure.tsThe Codex catalog window stays at 272k on purpose (input above that is billed at long-context rates); the server accepts up to ~1.05M, so the overflow test targets the hard limit.
Test plan
vp check,vp run typecheckpnpm test(246 unit)vp test test/e2e -t OpenAI(all OpenAI + Codex live suites, 342 tests)Regenerate the gitignored catalog before running the e2e suites:
🤖 Generated with Claude Code