Repository navigation
fix(aikit)!: correct thinking budget, response ceiling, and cost math - #25
Merged
Merged
Conversation
Every value aikit derives from model metadata is recomputed, and several were wrong in ways that silently changed what a request asked for or what it cost. Thinking budgets and ceilings: - An oversized budget consumed the answer, flooring it at a single token. A budget that does not fit is now reduced instead, always leaving 1024 tokens for the answer. - maxTokens now means the room reserved for the answer; the budget is added on top and the total clamped to the model ceiling. - Response ceilings are sized against the remaining context window rather than a fixed 32000-token guess. - A budget below a provider's own minimum is no longer sent as an enabled request. Thinking is dropped for that turn, so a nearly full context still produces an answer. - Anthropic requests counted the budget twice, inflating the ceiling and producing answers several times longer than an explicit maxTokens asked for. Reasoning configuration: - Reasoning is switched off explicitly when no level is requested. Models that reason by default were spending output tokens on reasoning the caller never asked for. - Newer Claude models receive an effort level and size each turn themselves rather than being pinned to a fixed budget. - Google gets per-family budgets, and a thinking level for the families that take one. - Thinking-token budgets now reach OpenAI-compatible endpoints that accept one, via compat.thinkingTokenBudgetField. Cost: - 1h cache writes were billed at the 5-minute rate, understating cache-write cost by roughly 37% whenever long retention is enabled. - Service-tier pricing applied only to Codex and used the requested tier rather than the tier the response was served at. Overflow detection: - Rate-limit and throttling errors were misread as context overflow, sending callers to compact a conversation that was never too long. - Added detection for providers that truncate an oversized prompt and stop with no output, and broadened the recognized provider error messages. Thinking resolution, request pricing, and context estimation move into llm/thinking.ts, llm/pricing.ts, and utils/estimate.ts so each derivation has one home and can be tested directly. BREAKING CHANGE: maxTokens now means the answer alone, so responses can be longer than before at the same setting. Default thinking budgets are absolute token counts (1024/2048/8192/16384) and are substantially smaller than the previous proportional defaults on models with a large output ceiling; set thinkingBudgets explicitly to keep the old amounts. Requests without a thinking level now disable reasoning explicitly, so models that reasoned by default no longer do so unless a level is set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fixes Gemini thoughtSignature issue
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every value aikit derives from model metadata is recomputed, and several were wrong in ways that silently changed what a request asked for or what it cost.
Thinking budgets and ceilings:
Reasoning configuration:
Cost:
Overflow detection:
Thinking resolution, request pricing, and context estimation move into llm/thinking.ts, llm/pricing.ts, and utils/estimate.ts so each derivation has one home and can be tested directly.
BREAKING CHANGE: maxTokens now means the answer alone, so responses can be longer than before at the same setting. Default thinking budgets are absolute token counts (1024/2048/8192/16384) and are substantially smaller than the previous proportional defaults on models with a large output ceiling; set thinkingBudgets explicitly to keep the old amounts. Requests without a thinking level now disable reasoning explicitly, so models that reasoned by default no longer do so unless a level is set.