Repository navigation
feat(mistralai): add chat completions API mode - #7381
Merged
tinalenguyen merged 5 commits intoSep 22, 2026
Merged
tinalenguyen merged 5 commits into
tinalenguyen merged 5 commits into
Conversation
Add `ApiMode` enum (`CONVERSATIONS` | `CHAT_COMPLETIONS`) to the Mistral plugin so callers can opt into the lower-latency Chat Completions API for sessions that only use function tools. Fix a bug in `_parse_completion_delta` where a redundant `if` guard caused tool-call arguments to be doubled on the first streamed delta. Remove dead `_emitted_tool_calls` / `_provider_tool_args` assignments from the chat completions path and add a test for provider-tool rejection.
|
|
leamoukarzel
marked this pull request as ready for review
September 21, 2026 15:31
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
tinalenguyen
approved these changes
Sep 22, 2026
2 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Mistral Chat Completions API (
/v1/chat/completions) delivers significantly lower p50 TTFT than the Conversations API (/v1/conversations): -21% on mistral-small (334 ms vs 421 ms) and -36% on mistral-medium (450 ms vs 707 ms).However, Chat Completions only supports function tools — provider tools (web search, document library, code interpreter, connectors) require the Conversations API. This PR adds an
api_modeswitch so callers can opt into the faster path when they don't need provider tools, while keeping Conversations as the default.Summary
Adds an
api_modeparameter to the MistralAI LLM plugin, allowing callers to choose between:ApiMode.CONVERSATIONS(default) — uses the Conversations API (/v1/conversations), which supports both function tools and provider tools (web search, document library, code interpreter, connectors).ApiMode.CHAT_COMPLETIONS— uses the Chat Completions API (/v1/chat/completions), which only supports function tools but has lower latency.The default remains
CONVERSATIONS, so this is fully backward-compatible.Changes
ApiModeenum exported fromlivekit.plugins.mistralaiLLM.__init__andupdate_optionsaccept anapi_modeparameterLLMStream._rundispatches to_run_conversations(existing logic, unchanged) or_run_chat_completions(new)ValueError)Test plan
TestChatCompletionsModetests cover text streaming, tool call accumulation, usage reporting, error mapping, and provider tool rejection