Skip to content

fix(aikit)!: correct thinking budget, response ceiling, and cost math - #25

Merged
sanchitrk merged 3 commits into
devfrom
fix-aikit-wire-protocol
Sep 5, 2026
Merged

sanchitrk merged 3 commits into
devfrom
fix-aikit-wire-protocol

Conversation

@sanchitrk

Copy link
Copy Markdown
Contributor

Every value aikit derives from model metadata is recomputed, and several were wrong in ways that silently changed what a request asked for or what it cost.

Thinking budgets and ceilings:

  • An oversized budget consumed the answer, flooring it at a single token. A budget that does not fit is now reduced instead, always leaving 1024 tokens for the answer.
  • maxTokens now means the room reserved for the answer; the budget is added on top and the total clamped to the model ceiling.
  • Response ceilings are sized against the remaining context window rather than a fixed 32000-token guess.
  • A budget below a provider's own minimum is no longer sent as an enabled request. Thinking is dropped for that turn, so a nearly full context still produces an answer.
  • Anthropic requests counted the budget twice, inflating the ceiling and producing answers several times longer than an explicit maxTokens asked for.

Reasoning configuration:

  • Reasoning is switched off explicitly when no level is requested. Models that reason by default were spending output tokens on reasoning the caller never asked for.
  • Newer Claude models receive an effort level and size each turn themselves rather than being pinned to a fixed budget.
  • Google gets per-family budgets, and a thinking level for the families that take one.
  • Thinking-token budgets now reach OpenAI-compatible endpoints that accept one, via compat.thinkingTokenBudgetField.

Cost:

  • 1h cache writes were billed at the 5-minute rate, understating cache-write cost by roughly 37% whenever long retention is enabled.
  • Service-tier pricing applied only to Codex and used the requested tier rather than the tier the response was served at.

Overflow detection:

  • Rate-limit and throttling errors were misread as context overflow, sending callers to compact a conversation that was never too long.
  • Added detection for providers that truncate an oversized prompt and stop with no output, and broadened the recognized provider error messages.

Thinking resolution, request pricing, and context estimation move into llm/thinking.ts, llm/pricing.ts, and utils/estimate.ts so each derivation has one home and can be tested directly.

BREAKING CHANGE: maxTokens now means the answer alone, so responses can be longer than before at the same setting. Default thinking budgets are absolute token counts (1024/2048/8192/16384) and are substantially smaller than the previous proportional defaults on models with a large output ceiling; set thinkingBudgets explicitly to keep the old amounts. Requests without a thinking level now disable reasoning explicitly, so models that reasoned by default no longer do so unless a level is set.

sanchitrk and others added 3 commits September 5, 2026 02:17
Every value aikit derives from model metadata is recomputed, and several were
wrong in ways that silently changed what a request asked for or what it cost.

Thinking budgets and ceilings:
- An oversized budget consumed the answer, flooring it at a single token.
  A budget that does not fit is now reduced instead, always leaving 1024
  tokens for the answer.
- maxTokens now means the room reserved for the answer; the budget is added
  on top and the total clamped to the model ceiling.
- Response ceilings are sized against the remaining context window rather
  than a fixed 32000-token guess.
- A budget below a provider's own minimum is no longer sent as an enabled
  request. Thinking is dropped for that turn, so a nearly full context still
  produces an answer.
- Anthropic requests counted the budget twice, inflating the ceiling and
  producing answers several times longer than an explicit maxTokens asked for.

Reasoning configuration:
- Reasoning is switched off explicitly when no level is requested. Models
  that reason by default were spending output tokens on reasoning the caller
  never asked for.
- Newer Claude models receive an effort level and size each turn themselves
  rather than being pinned to a fixed budget.
- Google gets per-family budgets, and a thinking level for the families that
  take one.
- Thinking-token budgets now reach OpenAI-compatible endpoints that accept
  one, via compat.thinkingTokenBudgetField.

Cost:
- 1h cache writes were billed at the 5-minute rate, understating cache-write
  cost by roughly 37% whenever long retention is enabled.
- Service-tier pricing applied only to Codex and used the requested tier
  rather than the tier the response was served at.

Overflow detection:
- Rate-limit and throttling errors were misread as context overflow, sending
  callers to compact a conversation that was never too long.
- Added detection for providers that truncate an oversized prompt and stop
  with no output, and broadened the recognized provider error messages.

Thinking resolution, request pricing, and context estimation move into
llm/thinking.ts, llm/pricing.ts, and utils/estimate.ts so each derivation has
one home and can be tested directly.

BREAKING CHANGE: maxTokens now means the answer alone, so responses can be
longer than before at the same setting. Default thinking budgets are absolute
token counts (1024/2048/8192/16384) and are substantially smaller than the
previous proportional defaults on models with a large output ceiling; set
thinkingBudgets explicitly to keep the old amounts. Requests without a
thinking level now disable reasoning explicitly, so models that reasoned by
default no longer do so unless a level is set.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fixes Gemini  thoughtSignature issue
@sanchitrk
sanchitrk merged commit 5a7af72 into dev Sep 5, 2026
@sanchitrk
sanchitrk deleted the fix-aikit-wire-protocol branch September 8, 2026 06:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant