Skip to content

feat(aikit): record thinking levels on assistant messages - #53

Merged
sanchitrk merged 4 commits into
canaryfrom
claude/cod-125-whenbusy-858cd8
Oct 9, 2026
Merged

sanchitrk merged 4 commits into
canaryfrom
claude/cod-125-whenbusy-858cd8

Conversation

@sanchitrk

Copy link
Copy Markdown
Contributor

Summary

Every assistant message now records two thinking fields, stamped by aikit at the wire:

  • thinkingLevel (required): the level the caller requested, before aikit fits it to the model. It is "off" when no level was requested.
  • providerThinkingLevel (optional): the provider-native value the request actually carried. For example, OpenAI minimal is recorded as low and OpenAI off as none. It is left empty for token-budget models and when the request carried no level.

providerThinkingLevel is read off the exact request body, not off the options aikit passes to the AI SDK, because the SDK drops OpenAI efforts for model ids it does not recognise:

  • For streamed requests, the body comes from the start-step part. streamText is now given include: { requestBody: true }.
  • For rejected requests, it comes from APICallError.requestBodyValues.

One reader covers every provider: reasoning.effort, reasoning_effort, Gemini thinkingConfig.thinkingLevel, and Anthropic output_config.effort unless thinking is disabled.

The harness needs no code change, because the message envelope is persisted whole. Messages stored before this change no longer decode, so local session databases need clearing.

Also in this PR

  • Codex fetch failures: a fetch that rejects before any response is now wrapped in a retryable APICallError carrying the request body, as AI SDK providers do. It is retried, aborts and timeouts pass through, and the failed message still records its effort.
  • Claude Haiku 5: treated as an adaptive thinking model, with xhigh and max mapped. Haiku 5.5 rejects fixed token budgets, takes every effort from low to max, and, unlike Opus and Sonnet 5.5, accepts disabled thinking.
  • Live test models:
    • Haiku 5.5 replaces claude-sonnet-4-5-20250929 and Fable 5.1 in the Anthropic suites.
    • Gemini 3.8 Flash leaves the OpenRouter image suite.
    • Haiku 5.5 skips the text-only placeholder test, which it declines to answer.

Test plan

  • vp check and vp run typecheck are clean.
  • Offline wire tests capture the real HTTP body and assert that the recorded providerThinkingLevel equals the effort on the wire. They cover clamped levels, off, caller overrides, token-budget and disabled thinking, and unknown OpenAI model ids.
  • Codex transport tests: a network failure is retried and records its effort, and an abort is not retried.
  • Live thinking-level suites assert both fields on real responses for OpenAI, Codex, Anthropic and OpenRouter.
  • Full live suites for aikit, harness and codework. The remaining failures come from the organisation's model access and rate limits, plus model variance that passed on rerun.

🤖 Generated with Claude Code

sanchitrk and others added 4 commits October 8, 2026 19:38
…ant messages

Every assistant message now records the thinking level the caller requested
(`thinkingLevel`, required, "off" when none) and the provider-native value the
request carried (`providerThinkingLevel`, optional).

The native value is read off the exact request body: the `start-step` part's
body for streamed requests, and `APICallError.requestBodyValues` for rejected
ones. Reading the options instead would record efforts the adapter drops, such
as OpenAI efforts for model ids the AI SDK does not know.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Codex fetch that rejects before any response now becomes a retryable
APICallError carrying the request body, as AI SDK providers do. The network
error is retried, aborts and timeouts pass through, and the failed message
still records the effort it sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Haiku 5.5 rejects a fixed thinking budget and takes every effort from low to
max, so it is sent adaptive thinking with xhigh and max mapped. Unlike Opus and
Sonnet 5.5 it accepts disabled thinking, so off stays available.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Haiku 5.5 is the cheap 5.x model in the Anthropic live suites; Fable added cost
without coverage Opus 5.5 does not already give. Gemini 3.8 Flash leaves the
OpenRouter image suite, and Haiku 5.5 skips the text-only placeholder test,
which it declines to answer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sanchitrk
sanchitrk merged commit 648af39 into canary Oct 9, 2026
@sanchitrk
sanchitrk deleted the claude/cod-125-whenbusy-858cd8 branch October 9, 2026 06:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant