Skip to content

Voice agents: cooperative interrupt, @pikku/voice-agents, and store-backed converse assertions - #1085

Merged
yasserf merged 5 commits into
mainfrom
spike/voice-agents
Aug 2, 2026
Merged

yasserf merged 5 commits into
mainfrom
spike/voice-agents

Conversation

@yasserf

@yasserf yasserf commented Aug 2, 2026 •

Copy link
Copy Markdown
Contributor

Spike: a Pikku agent you can hold a spoken conversation with, and a way to test that it actually did what it said.

Two halves, deliberately split so the OSS story stays clean: the API side lives in core, the browser side lives in a new @pikku/voice-agents package, and neither knows about the other's runtime.

What's here

core — cooperative interrupt. A verbal agent needs something a text one never did: the ability to be stopped mid-turn without losing what already happened. abortScope carries a cancellation signal down through the function runner, and beginChanges() gives an agent a checkpoint it can be interrupted at rather than anywhere. interrupted joins the AI run status enum, kept distinct from failed — the run was stopped on purpose, usually because the user talked over it, so it shouldn't surface in error reporting.

@pikku/voice-agents (new). Mic capture, speech and silence detection, playback queueing, spoken approvals.

@pikku/ai-deepinfra (new). STT/TTS provider. The voice path moved off Whisper once its invented filler turned out not to be filterable by confidence — it will produce a confident "Thank you." from silence.

cli — two fixes that meant actor.converse had never once worked under pikku scenario run.

Scenario steps execute in the CLI process, which stubbed its singleton services to three entries, so every conversing scenario died on AIProviderNotConfigured before saying anything. It now builds a runner the same way pikku dev does, and only when the project declares agents, so agent-less projects still need no AI env.

The second is worse because it is silent. supportsStructuredOutputs defaults to false, which drops outputSchema, degrades the call to bare json_object, and returns well-formed JSON with whatever keys the model felt like. Nothing errors; the caller reads undefined off every field it asked for and behaves as though the model said nothing.

Also adds scenarios.actorModel, so the model the actor thinks with is configurable and explicitly not the model under test.

The testing idea

A voice agent once replied "Okay — marking get lunch done." having called no tool at all, and on another run reached for deleteTodo. Both of those pass a test that only reads the actor's verdict, because the actor is judging the conversation, and the conversation was fine.

So the assertions go to the store instead: the todo exists, it is completed, exactly one of it exists, and the todo nobody mentioned is untouched.

Verified negatively — with completeTodo removed from the agent, the run produces:

Then the actor concludes the task succeeded                    ✓
Then the store holds the todo, completed                       ✗  No todo titled "get lunch" in the store

The actor's verdict passes while the store shows nothing happened. That's the case a verdict-only test ships green.

Note what is not asserted: which tools ran. The user asked for a todo to be done, not for completeTodo to be invoked. Whether the agent's sentence was honest about how it got there is a grade on the run rather than a check on the world, and belongs to the judge in #719.

Actor model

gpt-4.1-mini, chosen against a bench of the actor's four real behaviours — return each schema, and end the conversation when the goal is met. Every cheaper candidate failed the last one: Mistral-Small-3.2-24B scored 0/4 on "say you're done", so it burned the full 12-turn budget and took 320s instead of 33s. Per-token savings are an illusion there, because each wasted turn also bills the agent's model.

Verification

  • @pikku/core — 1591 pass, 0 fail
  • @pikku/ai-vercel 33, @pikku/ai-deepinfra 22, @pikku/voice-agents 28 — all passing
  • Both converse scenarios pass end-to-end against a live model
  • Rebased onto latest main; three conflicts resolved by hand (voice-output.ts for the new emit/signal params, scenario.ts alongside the new setRunSurface, and the func: → default: step rename), then the full suite re-run

One intentional behaviour change — please look at this

done now goes through the stream middleware rather than straight at the channel, and is awaited:

-    channel.send({ type: 'done' })
+    await outputChannel.send({ type: 'done' })

It is the only signal a stream hook gets that the reply is over, and hooks that buffer need it — voiceOutput speaks a trailing fragment that never reached a full stop, and waits on audio it has already paid to synthesize. Sent raw, that work is thrown away by the close() on the next line.

This changes observable middleware ordering, so verifiers/middleware-and-permissions is updated: modifyOutputStream now fires once more, after modifyOutput. That ordering is deliberate — modifyOutput is about the completed reply, done is about the stream carrying it — but it is a spec change and anyone with a modifyOutputStream hook will now see one extra event. Shout if that trade is wrong; the alternative is a separate flush hook that only buffering middleware implements.

The pre-push verifier suite is what caught this, not me.

Reviewer notes

  • e2e/pikku.config.json shows some unrelated array-formatting churn. That's lint-staged's prettier reformatting a file whose committed copy on main was stale — not a hand edit.
  • One pre-existing tsc error remains in packages/cli/src/services.ts (strictMeta), untouched by this branch and present on main.
  • This is a spike. packages/voice-agents has no README beyond a stub and the browser side has had far less adversarial testing than the core side.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added voice conversation support with microphone input, speech detection, silence handling, playback queues, barge-in, and spoken approvals.
    • Added AI agent interruption controls with interruption status, reasons, and delivery of pending tool results.
    • Added DeepInfra support for speech transcription and synthesis.
    • Added a complete voice assistant demonstration with todo-list interactions.
    • Added audio and interruption events for streamed agent responses.
  • Bug Fixes
    • Improved binary HTTP response handling and silent-transcription behavior.
    • Read-only tools are now clearly identified and protected from unintended mutations.
  • Documentation
    • Added setup and usage documentation for voice agents and DeepInfra integration.

yasserf and others added 5 commits August 2, 2026 18:20
…ware

Adds the primitive a verbal agent needs but a text one never did: the
ability to stop mid-turn without losing what already happened. `abortScope`
carries a cancellation signal down through the function runner, and
`beginChanges()` gives an agent a checkpoint it can be interrupted at
safely rather than anywhere.

`interrupted` joins the AI run status enum, kept distinct from `failed`:
the run was stopped on purpose, usually because the user talked over it,
so it should not surface in error reporting.

Also teaches `actor.converse` to say the word 'json' in its instructions.
Every call it makes wants a schema'd object back, and a gateway that
cannot take a JSON schema degrades to OpenAI's `json_object` mode, which
refuses the request outright unless the prompt mentions json. Saying it
once covers all three call sites.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DeepInfra fronts the STT/TTS models the voice path moved onto once
Whisper's invented filler turned out not to be filterable by confidence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The browser half of a verbal conversation — mic capture, speech and
silence detection, playback queueing and spoken approvals — kept as its
own package so the API half can stay in core and neither has to know
about the other's runtime.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…chemas

Two defects that meant `actor.converse` had never once worked under
`pikku scenario run`. Scenario steps execute in the CLI process, which
stubbed its singleton services to three entries — so every conversing
scenario died on AIProviderNotConfigured before saying anything. It now
builds a runner the same way `pikku dev` does, and only when the project
declares agents, so agent-less projects still need no AI env.

The second is worse because it is silent: `supportsStructuredOutputs`
defaults to false, which drops `outputSchema`, degrades the call to bare
`json_object`, and returns well-formed JSON with whatever keys the model
felt like. Nothing errors; the caller reads undefined off every field.

Adds `scenarios.actorModel` so the actor's model is configurable — it is
deliberately not the model under test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ence

A voice agent once replied 'Okay — marking get lunch done.' having called
no tool at all, and on another run reached for deleteTodo. Both pass a
test that only reads the actor's verdict, because the actor is judging
the conversation and the conversation was fine.

So the assertions go to the store instead: the todo exists, it is
completed, exactly one of it exists, and the todo nobody mentioned is
untouched. Verified negatively — with completeTodo removed from the
agent, the actor still returns passed:true and the store assertion is
what fails.

Note what is not asserted: which tools ran. The user asked for a todo to
be done, not for completeTodo to be invoked. Whether the agent's sentence
was honest about how it got there is a grade on the run rather than a
check on the world, and belongs to the judge in #719.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Aug 2, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 425f1f8

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Aug 2, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Changes

The PR adds a voice-agent package, DeepInfra speech services, interruptible AI-agent runs, voice streaming support, browser voice demos, scenario integration, and binary HTTP response handling.

Voice agent platform

Layer / File(s) Summary
Voice client lifecycle and interaction
packages/voice-agents/*
Adds microphone detection, voice sessions, ordered audio playback, spoken approvals, a React conversation hook, package exports, tests, and build configuration.

AI runtime and provider services

Layer / File(s) Summary
DeepInfra audio provider
packages/services/ai-deepinfra/*
Adds configurable transcription and speech models with request construction, response parsing, abort handling, tests, and package metadata.
Abort scopes and mutation checkpoints
packages/core/src/function/*, packages/core/src/types/*, packages/core/src/wirings/ai-agent/ai-agent.types.ts, packages/services/kysely/*
Adds ambient abort scopes, beginChanges(), readonly tool metadata, provider options, and the interrupted run status.
Interruptible agent execution
packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts, packages/core/src/wirings/ai-agent/ai-agent-runner.ts, packages/core/src/wirings/ai-agent/ai-agent-stream.ts
Adds run interruption, abort propagation, tool tracking, undelivered result persistence, interrupted stream state, resumed-run handling, and extensive tests.
Voice streams and interruption events
packages/core/src/wirings/ai-agent/voice-*, packages/core/src/wirings/ai-agent/ai-agent-agui.*, packages/services/ai-vercel/*, packages/core/src/wirings/rpc/*
Adds silent-input detection, script-aware synthesis, audio AG-UI events, interruption RPCs, wildcard providers, abort propagation, and undelivered tool-result conversion.

CLI and voice-agent integration

Layer / File(s) Summary
Scenario runner and provider wiring
packages/cli/src/functions/commands/*, packages/cli/types/config.d.ts, e2e/src/services.ts, e2e/package.json, e2e/pikku.config.json
Adds actor model configuration, development AI runner setup, wildcard mock routing, DeepInfra registration, and workspace dependencies.
Voice assistant demo flow
e2e/packages/functions/src/agents/*, e2e/packages/functions/src/functions/voice-demo.function.ts, e2e/packages/functions/src/wirings/voice-demo.wirings.ts, e2e/packages/todos/src/functions/*
Adds a spoken todo assistant, browser voice demo endpoints, speech and interruption handling, and readonly todo tools.
Voice scenario validation
e2e/packages/functions/tests/scenarios/*
Adds exact approval-reason assertions, completed-todo assertions, voice approval scenarios, and a live voice-assistant conversation scenario.

Transport and middleware validation

Layer / File(s) Summary
Binary HTTP response handling
packages/core/src/wirings/http/*
Allows HTTP responses to preserve binary bodies and validates binary and textual response behavior.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Browser
  participant VoiceSession
  participant voiceDemoRoutes
  participant AIStream
  participant AudioPlaybackQueue
  Browser->>VoiceSession: capture microphone turn
  VoiceSession->>voiceDemoRoutes: submit transcription or conversation input
  voiceDemoRoutes->>AIStream: start or interrupt agent run
  AIStream->>Browser: emit text, audio, or interruption events
  Browser->>AudioPlaybackQueue: enqueue audio chunks
  AudioPlaybackQueue->>Browser: report playback completion or heard text
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 17.39% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main changes: cooperative interruption, the new voice-agents package, and store-backed converse assertions.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch spike/voice-agents

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@yasserf
yasserf merged commit c3b5349 into main Aug 2, 2026
3 of 4 checks passed
@yasserf
yasserf deleted the spike/voice-agents branch August 2, 2026 16:53

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 20

🧹 Nitpick comments (23)
packages/core/src/wirings/http/web-request.test.ts (1)

354-369: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Cover the send branch for binary bodies.

createMockResponse does not define send. This test only validates res.arrayBuffer(body). applyWebResponse calls res.send(body) when an adapter provides it. Add a send-capable mock and assert the same byte sequence for that path.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/core/src/wirings/http/web-request.test.ts` around lines 354 - 369,
Extend the binary-body test around applyWebResponse to use a createMockResponse
variant that defines send, then assert send receives the original byte sequence
unchanged. Preserve the existing arrayBuffer assertion and verify the
adapter-provided send branch with the same binary payload.
e2e/packages/functions/src/functions/voice-demo.function.ts (2)

463-468: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Encoding byte by byte adds latency to every turn.

toBase64 concatenates one character per byte on the main thread. handleTurn awaits it before the transcription request starts, so the cost lands in the gap the page is trying to keep short. Chunked conversion removes the per-byte string growth.

♻️ Proposed change
 const toBase64 = async (blob) => {
   const bytes = new Uint8Array(await blob.arrayBuffer())
   let binary = ''
-  for (let i = 0; i < bytes.length; i++) binary += String.fromCharCode(bytes[i])
+  const CHUNK = 0x8000
+  for (let i = 0; i < bytes.length; i += CHUNK) {
+    binary += String.fromCharCode.apply(null, bytes.subarray(i, i + CHUNK))
+  }
   return btoa(binary)
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@e2e/packages/functions/src/functions/voice-demo.function.ts` around lines 463
- 468, Update toBase64 to avoid byte-by-byte string concatenation: convert the
Uint8Array to binary in chunks before calling btoa, preserving the same Base64
output and async behavior used by handleTurn.

144-148: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Bound the audio and text inputs.

audio and text accept a string of any length. Both routes are unauthenticated and both forward the payload to a billed provider. A single large request turns into a large provider charge, and Buffer.from on a very large base64 string allocates the whole decoded buffer in memory. Add a .max() to each schema. A few seconds of Opus is a few tens of kilobytes, so the bound can be generous and still useful.

♻️ Proposed change
 export const VoiceDemoTranscribeInput = z.object({
   /** The recorded turn, base64. JSON rather than a raw body so the route needs
    *  no binary handling on either side; a few seconds of Opus is small. */
-  audio: z.string(),
+  audio: z.string().max(4_000_000),
 })
 export const VoiceDemoSpeakInput = z.object({
-  text: z.string(),
+  text: z.string().max(2_000),
 })

Also applies to: 248-250

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@e2e/packages/functions/src/functions/voice-demo.function.ts` around lines 144
- 148, Bound the unauthenticated provider inputs by adding generous
maximum-length validation to the audio field in VoiceDemoTranscribeInput and the
text field in the corresponding text-input schema. Apply the limits to the
z.string() definitions before payloads reach Buffer.from or the billed provider,
preserving the existing schema behavior for valid short inputs.
e2e/packages/functions/src/agents/voice-assistant.agent.ts (1)

120-150: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Share the model ids and voice map with the demo route.

e2e/packages/functions/src/functions/voice-demo.function.ts redeclares the same values as ASR_MODEL, TTS_MODEL, and SPEAKABLE_SCRIPTS. Its comment states that these must stay in step with this agent. A copy in two files drifts silently, and drift here changes the voice the user hears or the script coverage check. Export the constants from one module and import them in both places.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@e2e/packages/functions/src/agents/voice-assistant.agent.ts` around lines 120
- 150, Centralize the voice configuration currently passed to voiceInput and
voiceOutput in this agent, including the ASR model id, TTS model id, and
speakableScripts map, as exported constants. Update both the voice assistant
agent and voice-demo.function.ts to import and use those shared constants,
removing the duplicate declarations while preserving the existing model and
voice values.
packages/voice-agents/README.md (1)

51-58: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add the missing exports to the reference table.

The table lists four exports, but the README itself demonstrates spokenApproval and interpretConsent imported from @pikku/voice-agents (line 113), and detectSpeech is documented at length elsewhere in the package as an alternative to detectSilence. None of these three appear in the table. Add rows for them so the table remains a complete reference.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/README.md` around lines 51 - 58, Update the “What each
piece is for” export table to include spokenApproval, interpretConsent, and
detectSpeech, with concise purposes matching their documented behavior and the
existing README terminology.
packages/voice-agents/run-tests.sh (1)

4-4: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value

Harden the shell script per shellcheck hints.

Shellcheck flags two spots:

  • Line 4: cd "$script_dir" has no failure guard (SC2164). If cd fails, the rest of the script runs from the wrong directory.
  • Line 35: files=($(find src -type f -name "*.test.ts")) splits on whitespace (SC2207). A test file path containing a space breaks into multiple array elements.

Both fixes are small and improve robustness.

🛠️ Proposed fixes
-cd "$script_dir"
+cd "$script_dir" || exit 1
-files=($(find src -type f -name "*.test.ts"))
+mapfile -t files < <(find src -type f -name "*.test.ts")

Also applies to: 35-35

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/run-tests.sh` at line 4, Harden the run-tests.sh setup
and file collection: update the cd command near script_dir to fail immediately
if changing directories fails, and replace the unquoted find-based array
assignment near files with whitespace-safe path handling so filenames containing
spaces remain single elements.

Source: Linters/SAST tools

packages/voice-agents/src/speech-detector.ts (1)

156-277: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Add test coverage for the two turn-detection modules. Both detectSpeech and detectSilence implement timing- and callback-sensitive logic (redemption windows, misfire/short-segment discarding, idempotent stop(), progress throttling) with no accompanying test file in this batch, while sibling modules in the same layer (audio-playback-queue.ts, spoken-approval.ts) each have one.

  • packages/voice-agents/src/speech-detector.ts#L156-L277: add speech-detector.test.ts covering onSpeechStart/onSpeechEnd/onVADMisfire handling, the redemptionMs timing adjustment in endTurn, and idempotent stop(), using a mocked VadModule/VadInstance.
  • packages/voice-agents/src/silence-detector.ts#L44-L118: add silence-detector.test.ts covering the audible/silent transition, onSilenceProgress throttling via PROGRESS_STEP, and idempotent stop(), using a mocked AudioContext/AudioNode/requestAnimationFrame.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/src/speech-detector.ts` around lines 156 - 277, Add
speech-detector.test.ts covering detectSpeech callbacks for speech starts,
speech ends, misfires, redemptionMs-adjusted timing, and idempotent stop() using
mocked VadModule/VadInstance; add silence-detector.test.ts covering
detectSilence audible-to-silent transitions, PROGRESS_STEP progress throttling,
and idempotent stop() using mocked AudioContext, AudioNode, and
requestAnimationFrame. Update only the tests; no direct production-code change
is required in either detector.
packages/voice-agents/package.json (1)

10-15: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add a types export condition for @pikku/voice-agents.

packages/voice-agents/package.json exposes import and require entrypoints, but consumers using exports need an explicit types entrypoint to resolve the emitted declarations. Add a top-level "types" field and list "types" first under "." in exports.

📦 Proposed fix for the export map
+  "types": "./dist/esm/index.d.ts",
   "exports": {
     ".": {
+      "types": "./dist/esm/index.d.ts",
       "import": "./dist/esm/index.js",
       "require": "./dist/cjs/index.js"
     }
   },
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/package.json` around lines 10 - 15, Add the
package-level types field pointing to the emitted declarations, and update the
"." export map in the package configuration to list a types condition first
before import and require, using the package’s existing declaration output path.
packages/voice-agents/tsconfig.cjs.json (1)

5-5: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Replace deprecated moduleResolution: "node" in the CJS tsconfig.

TypeScript 6 marks moduleResolution: "node"/"node10" as deprecated in favor of moduleResolution: "node16"/"node18"/NodeNext. This config uses commonjs but currently sets moduleResolution: "node", so change both CJK entries to moduleResolution: "node16" and remove ignoreDeprecations: "6.0".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/tsconfig.cjs.json` at line 5, Update
packages/voice-agents/tsconfig.cjs.json to replace both moduleResolution: "node"
entries with moduleResolution: "node16", and remove the ignoreDeprecations:
"6.0" setting while preserving the existing CommonJS module configuration.
packages/voice-agents/src/spoken-approval.ts (1)

157-170: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

The ReDoS hint here is a false positive, but you can precompile the patterns.

phrases comes from the module-level GRANTS and DENIALS constants. No user input reaches new RegExp. The static analysis warning does not apply. takePhrases still builds 37 RegExp objects on every call. Build them once at module scope.

♻️ Proposed refactor
-const takePhrases = (text: string, phrases: string[]) => {
+const patternsFor = (phrases: string[]) =>
+  phrases.map((phrase) => new RegExp(`\\b${phrase}\\b`, 'g'))
+
+const GRANT_PATTERNS = patternsFor(GRANTS)
+const DENIAL_PATTERNS = patternsFor(DENIALS)
+
+const takePhrases = (text: string, patterns: RegExp[]) => {
   let found = false
   let remaining = text
-  for (const phrase of phrases) {
+  for (const pattern of patterns) {
     // Matched phrases are removed so a later pass cannot read the same words
     // again — this is what keeps "no problem" from also counting as a refusal.
-    const next = remaining.replace(new RegExp(`\\b${phrase}\\b`, 'g'), ' ')
+    pattern.lastIndex = 0
+    const next = remaining.replace(pattern, ' ')
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/src/spoken-approval.ts` around lines 157 - 170,
Precompile the phrase-matching regular expressions for the module-level GRANTS
and DENIALS values instead of constructing them inside takePhrases on every
call. Update takePhrases to reuse the prebuilt patterns while preserving phrase
removal order and the existing found/remaining behavior.

Source: Linters/SAST tools

packages/voice-agents/src/audio-playback-queue.ts (2)

86-93: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Document that the returned promise rejects on decode failure.

drain() handles a failed decode internally, but enqueue returns a derived promise from buffer.then(...). A caller that writes void queue.enqueue(...) gets an unhandled rejection. The test file already works around this at audio-playback-queue.test.ts Line 97. State the contract in the JSDoc, so callers know they must handle rejection.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/src/audio-playback-queue.ts` around lines 86 - 93,
Update the JSDoc for enqueue in the audio playback queue to explicitly document
that its returned promise resolves when decoding completes but rejects if audio
decoding fails. Keep the existing enqueue and drain behavior unchanged, and make
clear that callers must handle the rejection.

114-132: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

An interrupt during playback leaves the play() promise pending forever.

interrupt() sets source.onended = null and calls source.stop(). The Promise created in play() then has no path to resolve. The drain() loop that awaits it stays suspended permanently and retains its closure. Each interruption during playback adds one such suspended loop.

Keep a resolver for the current utterance and settle it in interrupt().

♻️ Proposed refactor
   private current: {
     text: string
     source: AudioBufferSourceNode
     startedAt: number
     duration: number
+    done: () => void
   } | null = null
-      const { text, source, startedAt, duration } = this.current
+      const { text, source, startedAt, duration, done } = this.current
       complete = false
       source.onended = null
       try {
         source.stop()
       } catch {
         // Already stopped by the scheduler; nothing to unwind.
       }
       const elapsed = Math.max(0, this.ctx.currentTime - startedAt)
       const fraction = duration > 0 ? Math.min(1, elapsed / duration) : 0
       const cut = text.slice(0, Math.round(text.length * fraction)).trimEnd()
       if (cut) heard.push(cut)
       this.current = null
+      done()
       this.current = {
         text: pending.text,
         source,
         startedAt: this.ctx.currentTime,
         duration: pending.buffer.duration,
+        done: resolve,
       }

Also applies to: 180-200

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/src/audio-playback-queue.ts` around lines 114 - 132,
Update the current utterance state in play() to retain its promise resolver,
then invoke that resolver from interrupt() before clearing this.current so the
interrupted playback promise settles and drain() can continue. Preserve normal
onended resolution and ensure the resolver is settled only once.
packages/voice-agents/src/use-voice-conversation.ts (1)

81-82: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Move the optionsRef.current write into an effect.

This hook declares React 18/19 support, and mutating a ref during render can leave the ref pointing to an abandoned render in concurrent mode. Set optionsRef.current = options in useEffect, and add the necessary cleanup so start/onSpeechStart do not use stale options after unmount or dependency change.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/src/use-voice-conversation.ts` around lines 81 - 82,
Move the optionsRef.current assignment out of render and into a useEffect within
the voice conversation hook. Add effect cleanup that invalidates or clears the
ref on unmount and before dependency changes, ensuring start and onSpeechStart
cannot read stale options.
packages/services/ai-deepinfra/run-tests.sh (1)

31-35: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Use a null-delimited read to collect test files.

Word splitting on find output breaks on paths that contain spaces or globbing characters. mapfile with -print0 removes that risk. Note also that shopt -s nullglob on Line 7 and the pattern variable are now only decorative, because find performs the discovery.

Static analysis reports the same concern (SC2207).

♻️ Proposed change
-# Expand the pattern into an array of files
-files=($(find src -type f -name "*.test.ts"))
+# Expand the pattern into an array of files
+files=()
+while IFS= read -r -d '' file; do
+  files+=("$file")
+done < <(find src -type f -name "*.test.ts" -print0)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/services/ai-deepinfra/run-tests.sh` around lines 31 - 35, Update the
test-file collection in the script to read null-delimited results from find
using mapfile and find -print0, preserving paths containing spaces or glob
characters. Remove the now-unused pattern variable and nullglob setting because
discovery no longer relies on shell glob expansion.

Source: Linters/SAST tools

packages/services/ai-deepinfra/src/transcription-model.ts (2)

179-183: 🗄️ Data Integrity & Integration | 🔵 Trivial | 💤 Low value

String(value) flattens object and array provider extras.

A provider extra such as { deepinfra: { diarization: { speakers: 2 } } } becomes the multipart value [object Object]. The speech model sends extras as JSON, so the two models accept different extra shapes for the same option name. Serialise non-primitive values as JSON.

♻️ Proposed change
     for (const [key, value] of Object.entries(
       providerExtras(options.providerOptions)
     )) {
-      if (value !== undefined && value !== null) body.append(key, String(value))
+      if (value === undefined || value === null) continue
+      body.append(
+        key,
+        typeof value === 'object' ? JSON.stringify(value) : String(value)
+      )
     }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/services/ai-deepinfra/src/transcription-model.ts` around lines 179 -
183, Update the provider extras loop in the transcription model to serialize
object and array values with JSON while preserving direct string conversion for
primitive values. Ensure multipart body fields receive valid JSON for
non-primitive extras, including nested objects such as diarization
configuration.

203-204: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Give a non-JSON success response the same model-named error as a failure.

response.json() throws a bare SyntaxError when a 200 response carries HTML or plain text, for example from a proxy or an edge cache. The failure path on Line 201 names the model, and speech-model.ts guards on content-type before it parses. This path does neither, so the caller loses the model id and the body.

♻️ Proposed change
-    const payload = (await response.json()) as DeepInfraTranscriptionResponse
+    const raw = await response.text()
+    let payload: DeepInfraTranscriptionResponse
+    try {
+      payload = JSON.parse(raw) as DeepInfraTranscriptionResponse
+    } catch {
+      throw new Error(
+        `DeepInfra transcription model '${this.modelId}' returned a non-JSON body: ${raw.slice(0, 500)}`
+      )
+    }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/services/ai-deepinfra/src/transcription-model.ts` around lines 203 -
204, Update the transcription response handling around response.json() to
validate the response content type before parsing, and handle non-JSON success
bodies with the same model-named error format as the existing failure path.
Preserve the parsed payload flow for valid JSON responses while including the
response body and model identifier when rejecting invalid success responses.
packages/services/ai-deepinfra/package.json (1)

20-25: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Align the release script with Yarn and consider exporting types in the exports map.

Two small points:

  • Line 20 uses npm run build && npm test while the rest of the monorepo uses Yarn.
  • The exports map has no types condition. Consumers that resolve with node16/nodenext or bundler read exports and can miss dist/index.d.ts.
♻️ Proposed change
-    "release": "npm run build && npm test",
+    "release": "yarn build && yarn test",
     "prepublishOnly": "yarn build"
   },
   "exports": {
-    ".": "./dist/index.js"
+    ".": {
+      "types": "./dist/index.d.ts",
+      "default": "./dist/index.js"
+    }
   },
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/services/ai-deepinfra/package.json` around lines 20 - 25, Update the
package.json release script to use the repository’s Yarn commands instead of
npm, while preserving the existing build-and-test sequence. Extend the exports
entry for "." with a types condition pointing to the generated declaration file
at dist/index.d.ts alongside the existing runtime export.
packages/services/ai-deepinfra/src/deepinfra-config.ts (2)

39-47: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

A differently cased authorization key in settings.headers produces two header entries.

The returned object is a plain record. If a caller passes headers: { Authorization: '…' }, the record holds both authorization and Authorization. fetch then normalises both names into one field and joins the values with a comma, which yields an invalid credential rather than an override.

Normalise the key names before the merge, or document that settings.headers must use lowercase names.

♻️ Proposed change
   headers: () => {
     const apiKey = settings.apiKey ?? process.env.DEEPINFRA_API_KEY
     if (!apiKey) {
       throw new Error(
         'DeepInfra API key is missing. Pass `apiKey` to createDeepInfra(), or set DEEPINFRA_API_KEY.'
       )
     }
-    return { authorization: `Bearer ${apiKey}`, ...settings.headers }
+    const extra = Object.fromEntries(
+      Object.entries(settings.headers ?? {}).map(([key, value]) => [
+        key.toLowerCase(),
+        value,
+      ])
+    )
+    return { authorization: `Bearer ${apiKey}`, ...extra }
   },
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/services/ai-deepinfra/src/deepinfra-config.ts` around lines 39 - 47,
Update the headers callback in the DeepInfra configuration so settings.headers
keys are normalised case-insensitively before merging with the generated
authorization header, ensuring any casing of authorization overrides or is
overridden without producing duplicate entries. Preserve all other custom
headers unchanged.

48-48: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Bind the default fetch to globalThis.

settings.fetch ?? globalThis.fetch stores a detached reference. Browsers reject detached fetch calls and throw TypeError: Illegal invocation, so the browser-facing @pikku/voice-agents path can fail. Resolve globalThis.fetch lazily and bind it.

♻️ Proposed change
-  fetch: settings.fetch ?? globalThis.fetch,
+  fetch: settings.fetch ?? ((...args) => globalThis.fetch(...args)),
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/services/ai-deepinfra/src/deepinfra-config.ts` at line 48, Update
the default fetch handling in the deepinfra configuration so that when
settings.fetch is absent, globalThis.fetch is resolved lazily and bound to
globalThis before storage or invocation; preserve any explicitly supplied
settings.fetch unchanged.
packages/services/ai-deepinfra/src/transcription-model.test.ts (1)

5-22: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

stubFetch is duplicated in both test files.

speech-model.test.ts Lines 5-24 defines a near-identical helper that adds a Uint8Array branch. Extract one shared helper, for example src/test-fetch.ts, so both models test against the same stub behaviour. The tsconfig.json exclude list only removes **/*.test.ts, so name any shared helper accordingly or add it to the exclude list to keep it out of dist.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/services/ai-deepinfra/src/transcription-model.test.ts` around lines
5 - 22, Extract the duplicated stubFetch helper from transcription-model.test.ts
and speech-model.test.ts into one shared test-fetch helper, preserving the
Uint8Array, string, JSON, status, headers, and call-capture behavior used by
both tests. Import and reuse that helper in both test files, and name the shared
file appropriately or update tsconfig.json so it remains excluded from dist.
packages/services/ai-deepinfra/src/speech-model.test.ts (1)

101-143: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider covering abortSignal and per-call header merging for the speech model.

transcription-model.test.ts verifies that abortSignal reaches fetch on Lines 227-239. The speech model forwards the same signal on speech-model.ts Line 78, and it merges definedHeaders(options.headers) on Line 60, but neither path has a test here. Barge-in depends on the signal, so a regression would be silent.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/services/ai-deepinfra/src/speech-model.test.ts` around lines 101 -
143, Add speech-model tests covering both per-call abortSignal propagation to
fetch and per-call header merging with the configured headers. Use the existing
stubFetch and speech model setup, assert the fetch receives the exact signal,
and verify custom defined headers are included alongside the model’s default
headers.
packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts (1)

289-319: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Remove the call.mutating flag or make it change state.

call.mutating is initialised to true at Line 295, and onBeginChanges at Line 303-305 sets it to true again. The collectResult getter at Line 312-314 therefore always returns true. The flag and the hook have no effect.

The conservative default is correct and is documented in beginChanges. The suppression for a tool that stops at its checkpoint already comes from the AbandonedError branch at Line 179. Drop the flag so the code states what it does.

♻️ Proposed simplification
       execute: (input: unknown) => {
         // Assumed to have changed something unless it says otherwise — the tool
         // that never declares its checkpoint is the one that cannot be assumed
         // harmless. `beginChanges()` can only ever make this more precise.
-        const call = { mutating: true }
         const scope: AbortScope = {
           get abandoned() {
             return handle.signal.aborted
           },
           get reason() {
             return handle.interruption?.reason
           },
-          onBeginChanges: () => {
-            call.mutating = true
-          },
         }
         return handle.trackTool(
           tool.name,
           randomUUID(),
-          () => runInAbortScope(scope, () => tool.execute(input)),
-          {
-            get collectResult() {
-              return call.mutating
-            },
-          }
+          () => runInAbortScope(scope, () => tool.execute(input))
         )
       },
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts` around lines 289 -
319, Remove the redundant call.mutating state and the onBeginChanges hook from
the execute wrapper in the tool mapping. Keep collectResult as an unconditional
true value, preserving the documented conservative default and existing
AbandonedError suppression behavior.
packages/core/src/wirings/ai-agent/voice-output.ts (1)

252-261: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Do not log an aborted synthesis as an error.

On barge-in the run aborts, and synthesizeAudio rejects for every sentence still in flight. Each rejection reaches this branch and writes a logger.error line. A normal interruption then produces error-level noise proportional to the number of queued sentences. Distinguish the abort from a provider failure.

♻️ Proposed change
         state.tail = (state.tail ?? Promise.resolve()).then(async () => {
           const result = await synthesis
           if (!result.ok) {
+            // An interrupt cancels every sentence still in flight. That is the
+            // feature working, not a provider failure.
+            if (signal?.aborted) return
             // One sentence going unspoken is better than the rest of the reply
             // never arriving, so the chain carries on.
             logger?.error(
               `voiceOutput could not synthesize a sentence: ${String(result.error)}`
             )
             return
           }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/core/src/wirings/ai-agent/voice-output.ts` around lines 252 - 261,
Update the error handling around the synthesis result in the state.tail chain to
distinguish an expected run abort from a provider failure. Suppress logger.error
for abort-related rejections during barge-in, while preserving the existing
error log and chain continuation for genuine synthesis failures.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@e2e/packages/functions/src/agents/voice.agent.ts`:
- Around line 9-13: Resolve the model inconsistency in the voice agent’s
aiMiddleware configuration: use deepinfra/openai/whisper-large-v3-turbo as the
primary voiceInput model unless Nemotron is intentionally required here. If
retaining Nemotron, add a brief comment explaining why voice.agent differs from
voice-assistant.agent and voice-demo.function.

In `@e2e/packages/functions/src/functions/voice-demo.function.ts`:
- Around line 748-755: Guard inFlightTools before accessing its length in the
interrupt handling logic after post('/voice-demo/interrupt'). Preserve the
existing note behavior when stopped is true and tools are present, while
treating an omitted or undefined inFlightTools value as an empty list.
- Around line 880-885: Update the startSession error path around the shown catch
block to destroy the existing AudioPlaybackQueue before clearing playback.
Invoke playback.destroy() when the queue was created, then set playback to null
while preserving the existing session reset and idle-state behavior.
- Around line 304-318: The voiceDemoInterrupt route currently claims
rpc.agent.interrupt performs caller ownership validation despite being
sessionless with auth disabled. Update the route to establish and pass the demo
owner/principal before calling rpc.agent.interrupt, or restrict interruption to
runs verified as owned during stream runtime; otherwise remove the inaccurate
ownership assertion from the function comment.
- Around line 44-93: Update the asset resolution used with SERVABLE and distOf
so each package group resolves from the correct asset directory rather than
assuming the package entry-point directory. Use package-specific resolve
targets: the voice assets from the voice package, vad-web’s root-level bundle
assets from its package root, and onnxruntime-web’s listed runtime assets from
its dist directory; preserve the existing allow-list and clear failure behavior
for unresolved assets.

In `@e2e/packages/functions/tests/scenarios/converse.steps.ts`:
- Around line 131-155: Update the matches filter in the todo assertion to use
case-insensitive exact title equality instead of substring matching. Preserve
the existing no-match and duplicate diagnostics, ensuring titles such as “get
lunch ingredients” do not match “get lunch.”

In `@packages/core/src/function/function-runner.ts`:
- Around line 294-299: Update the invocation setup around
invocationWire.beginChanges to capture its previous value, assign it only for
non-readonly functions, and delete it for readonly functions. Restore the
original beginChanges value in both existing finally blocks, following the
handling of functionId, audit, and addonNamespace, so reused wires retain their
prior state.

In `@packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts`:
- Around line 331-346: Update trackInterruptNote so the tracked promise handles
write failures internally: catch rejections from write/finally, log the error
using the module’s established logging mechanism, and ensure the promise stored
in pendingNotes resolves after logging. Preserve the existing cleanup that
removes the entry only when it still matches the tracked promise, so
awaitPendingInterruptNote does not propagate storage failures into the next run.

In `@packages/core/src/wirings/ai-agent/ai-agent-prepare.ts`:
- Around line 1027-1031: Update the pending interrupt-note wait in
prepareAgentRun to catch failures from awaitPendingInterruptNote(threadId), log
them as note-persistence failures, and allow preparation to continue so
streamAIAgent/runAIAgent retain their normal error/done handling. Keep
successful waiting behavior unchanged.

In `@packages/core/src/wirings/ai-agent/ai-agent-runner.ts`:
- Around line 381-396: Preserve accumulated usage when interrupted in both run
paths. In packages/core/src/wirings/ai-agent/ai-agent-runner.ts at lines 381-396
and 861-874, hoist totalUsage above the try blocks in runAIAgent and
continueAfterToolResultSync, then include usage with totalUsage and agent.model
in each interrupted updateRun call; apply the same change at both sites.

In `@packages/core/src/wirings/ai-agent/ai-agent.types.ts`:
- Around line 63-79: Update all AI storage implementations used by
convertToSDKMessages so saveMessages persists the interrupted and undelivered
fields, and getMessages restores them on returned messages. Preserve these
optional flags unchanged across writes and reads, including false/undefined
behavior, so assistant and tool messages retain their interrupt metadata.

In `@packages/core/src/wirings/ai-agent/voice-input.ts`:
- Around line 130-139: Update the voice listening loop around
streamAIAgent/runAIAgent to catch NoSpeechDetectedError from
voiceInput.modifyInput, resume listening without forwarding the error to the
agent transport or reporting a failed run, and preserve existing handling for
all other errors.

In `@packages/core/src/wirings/ai-agent/voice-output.ts`:
- Around line 270-282: Bound the await on state.tail in the event.type ===
'done' branch so a stalled synthesis call cannot prevent completion
indefinitely. Race the pending tail against a finite deadline and continue
clearing state.tail and returning the audio-done event when the deadline
expires, while preserving the existing behavior when the tail settles normally.
- Around line 43-54: Update the SCRIPT_RANGES.latin pattern to match only Latin
script characters, excluding punctuation and symbols currently covered by the
broad A–ɏ range. Prefer a Unicode Script-based expression if supported, while
preserving the behavior of the other script patterns.

In `@packages/services/ai-vercel/src/vercel-ai-agent-runner.ts`:
- Around line 9-16: Add `@ai-sdk/provider` as a direct dependency or
peerDependency in packages/services/ai-vercel/package.json, matching the
existing import of SharedV3ProviderOptions in vercel-ai-agent-runner.ts and
using a compatible version with the installed ai package.

In `@packages/voice-agents/src/audio-playback-queue.ts`:
- Around line 140-153: Update destroy() and the interrupt/resume flow so the
asynchronous resume triggered by interrupt() cannot become an unhandled
rejection when ctx.close() runs; either await and handle the resume before
closing the AudioContext or attach an explicit rejection handler, while
preserving the queue’s existing interruption behavior.

In `@packages/voice-agents/src/speech-detector.ts`:
- Around line 262-277: Update the stop method returned by the speech detector
setup to attach a rejection handler to vad.destroy(). Route any destruction
error through handlers.onError, preserving the existing stopped guard and
fire-and-forget behavior.

In `@packages/voice-agents/src/spoken-approval.ts`:
- Around line 131-147: Update interpretConsent and its phrase matching around
DENIALS/takePhrases so negated grant phrases such as “not okay” and “that is not
fine” return “denied” rather than “granted”; detect “not” before a grant phrase
or include it in the denial handling, and add the specified covering assertions
to spoken-approval.test.ts.

In `@packages/voice-agents/src/voice-session.ts`:
- Around line 21-24: Update the silenceDelay documentation in the voice session
options to state the actual default of 700 milliseconds, matching the fallback
used by captureNextTurn; leave the minDecibels documentation unchanged.
- Around line 134-150: Ensure the start() flow arms the speech detector at most
once per acquired stream by guarding startSpeechDetector() with the existing
detector state, such as this.stopSpeech. Preserve detector reuse across stop()
followed by start(), while still starting it when no detector has been created.

---

Nitpick comments:
In `@e2e/packages/functions/src/agents/voice-assistant.agent.ts`:
- Around line 120-150: Centralize the voice configuration currently passed to
voiceInput and voiceOutput in this agent, including the ASR model id, TTS model
id, and speakableScripts map, as exported constants. Update both the voice
assistant agent and voice-demo.function.ts to import and use those shared
constants, removing the duplicate declarations while preserving the existing
model and voice values.

In `@e2e/packages/functions/src/functions/voice-demo.function.ts`:
- Around line 463-468: Update toBase64 to avoid byte-by-byte string
concatenation: convert the Uint8Array to binary in chunks before calling btoa,
preserving the same Base64 output and async behavior used by handleTurn.
- Around line 144-148: Bound the unauthenticated provider inputs by adding
generous maximum-length validation to the audio field in
VoiceDemoTranscribeInput and the text field in the corresponding text-input
schema. Apply the limits to the z.string() definitions before payloads reach
Buffer.from or the billed provider, preserving the existing schema behavior for
valid short inputs.

In `@packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts`:
- Around line 289-319: Remove the redundant call.mutating state and the
onBeginChanges hook from the execute wrapper in the tool mapping. Keep
collectResult as an unconditional true value, preserving the documented
conservative default and existing AbandonedError suppression behavior.

In `@packages/core/src/wirings/ai-agent/voice-output.ts`:
- Around line 252-261: Update the error handling around the synthesis result in
the state.tail chain to distinguish an expected run abort from a provider
failure. Suppress logger.error for abort-related rejections during barge-in,
while preserving the existing error log and chain continuation for genuine
synthesis failures.

In `@packages/core/src/wirings/http/web-request.test.ts`:
- Around line 354-369: Extend the binary-body test around applyWebResponse to
use a createMockResponse variant that defines send, then assert send receives
the original byte sequence unchanged. Preserve the existing arrayBuffer
assertion and verify the adapter-provided send branch with the same binary
payload.

In `@packages/services/ai-deepinfra/package.json`:
- Around line 20-25: Update the package.json release script to use the
repository’s Yarn commands instead of npm, while preserving the existing
build-and-test sequence. Extend the exports entry for "." with a types condition
pointing to the generated declaration file at dist/index.d.ts alongside the
existing runtime export.

In `@packages/services/ai-deepinfra/run-tests.sh`:
- Around line 31-35: Update the test-file collection in the script to read
null-delimited results from find using mapfile and find -print0, preserving
paths containing spaces or glob characters. Remove the now-unused pattern
variable and nullglob setting because discovery no longer relies on shell glob
expansion.

In `@packages/services/ai-deepinfra/src/deepinfra-config.ts`:
- Around line 39-47: Update the headers callback in the DeepInfra configuration
so settings.headers keys are normalised case-insensitively before merging with
the generated authorization header, ensuring any casing of authorization
overrides or is overridden without producing duplicate entries. Preserve all
other custom headers unchanged.
- Line 48: Update the default fetch handling in the deepinfra configuration so
that when settings.fetch is absent, globalThis.fetch is resolved lazily and
bound to globalThis before storage or invocation; preserve any explicitly
supplied settings.fetch unchanged.

In `@packages/services/ai-deepinfra/src/speech-model.test.ts`:
- Around line 101-143: Add speech-model tests covering both per-call abortSignal
propagation to fetch and per-call header merging with the configured headers.
Use the existing stubFetch and speech model setup, assert the fetch receives the
exact signal, and verify custom defined headers are included alongside the
model’s default headers.

In `@packages/services/ai-deepinfra/src/transcription-model.test.ts`:
- Around line 5-22: Extract the duplicated stubFetch helper from
transcription-model.test.ts and speech-model.test.ts into one shared test-fetch
helper, preserving the Uint8Array, string, JSON, status, headers, and
call-capture behavior used by both tests. Import and reuse that helper in both
test files, and name the shared file appropriately or update tsconfig.json so it
remains excluded from dist.

In `@packages/services/ai-deepinfra/src/transcription-model.ts`:
- Around line 179-183: Update the provider extras loop in the transcription
model to serialize object and array values with JSON while preserving direct
string conversion for primitive values. Ensure multipart body fields receive
valid JSON for non-primitive extras, including nested objects such as
diarization configuration.
- Around line 203-204: Update the transcription response handling around
response.json() to validate the response content type before parsing, and handle
non-JSON success bodies with the same model-named error format as the existing
failure path. Preserve the parsed payload flow for valid JSON responses while
including the response body and model identifier when rejecting invalid success
responses.

In `@packages/voice-agents/package.json`:
- Around line 10-15: Add the package-level types field pointing to the emitted
declarations, and update the "." export map in the package configuration to list
a types condition first before import and require, using the package’s existing
declaration output path.

In `@packages/voice-agents/README.md`:
- Around line 51-58: Update the “What each piece is for” export table to include
spokenApproval, interpretConsent, and detectSpeech, with concise purposes
matching their documented behavior and the existing README terminology.

In `@packages/voice-agents/run-tests.sh`:
- Line 4: Harden the run-tests.sh setup and file collection: update the cd
command near script_dir to fail immediately if changing directories fails, and
replace the unquoted find-based array assignment near files with whitespace-safe
path handling so filenames containing spaces remain single elements.

In `@packages/voice-agents/src/audio-playback-queue.ts`:
- Around line 86-93: Update the JSDoc for enqueue in the audio playback queue to
explicitly document that its returned promise resolves when decoding completes
but rejects if audio decoding fails. Keep the existing enqueue and drain
behavior unchanged, and make clear that callers must handle the rejection.
- Around line 114-132: Update the current utterance state in play() to retain
its promise resolver, then invoke that resolver from interrupt() before clearing
this.current so the interrupted playback promise settles and drain() can
continue. Preserve normal onended resolution and ensure the resolver is settled
only once.

In `@packages/voice-agents/src/speech-detector.ts`:
- Around line 156-277: Add speech-detector.test.ts covering detectSpeech
callbacks for speech starts, speech ends, misfires, redemptionMs-adjusted
timing, and idempotent stop() using mocked VadModule/VadInstance; add
silence-detector.test.ts covering detectSilence audible-to-silent transitions,
PROGRESS_STEP progress throttling, and idempotent stop() using mocked
AudioContext, AudioNode, and requestAnimationFrame. Update only the tests; no
direct production-code change is required in either detector.

In `@packages/voice-agents/src/spoken-approval.ts`:
- Around line 157-170: Precompile the phrase-matching regular expressions for
the module-level GRANTS and DENIALS values instead of constructing them inside
takePhrases on every call. Update takePhrases to reuse the prebuilt patterns
while preserving phrase removal order and the existing found/remaining behavior.

In `@packages/voice-agents/src/use-voice-conversation.ts`:
- Around line 81-82: Move the optionsRef.current assignment out of render and
into a useEffect within the voice conversation hook. Add effect cleanup that
invalidates or clears the ref on unmount and before dependency changes, ensuring
start and onSpeechStart cannot read stale options.

In `@packages/voice-agents/tsconfig.cjs.json`:
- Line 5: Update packages/voice-agents/tsconfig.cjs.json to replace both
moduleResolution: "node" entries with moduleResolution: "node16", and remove the
ignoreDeprecations: "6.0" setting while preserving the existing CommonJS module
configuration.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 07cbdd15-a1a3-478e-903a-f2d8953a990e

📥 Commits

Reviewing files that changed from the base of the PR and between c984df6 and 425f1f8.

⛔ Files ignored due to path filters (1)
  • yarn.lock is excluded by !**/yarn.lock, !**/*.lock
📒 Files selected for processing (75)
  • e2e/package.json
  • e2e/packages/functions/package.json
  • e2e/packages/functions/src/agents/voice-assistant.agent.ts
  • e2e/packages/functions/src/agents/voice.agent.ts
  • e2e/packages/functions/src/functions/voice-demo.function.ts
  • e2e/packages/functions/src/wirings/voice-demo.wirings.ts
  • e2e/packages/functions/tests/scenarios/agent-assert.steps.ts
  • e2e/packages/functions/tests/scenarios/converse.steps.ts
  • e2e/packages/functions/tests/scenarios/voice-approval.feature.ts
  • e2e/packages/functions/tests/scenarios/voice-assistant-converse.feature.ts
  • e2e/packages/todos/src/functions/getTodo.function.ts
  • e2e/packages/todos/src/functions/listTodos.function.ts
  • e2e/pikku.config.json
  • e2e/src/services.ts
  • packages/cli/src/functions/commands/dev-ai-runner.ts
  • packages/cli/src/functions/commands/scenario.ts
  • packages/cli/types/config.d.ts
  • packages/core/src/function/abort-scope.test.ts
  • packages/core/src/function/abort-scope.ts
  • packages/core/src/function/function-runner.ts
  • packages/core/src/index.ts
  • packages/core/src/services/ai-agent-runner-service.ts
  • packages/core/src/types/core.types.ts
  • packages/core/src/wirings/actor-flow/run-conversation.ts
  • packages/core/src/wirings/ai-agent/ai-agent-agui.test.ts
  • packages/core/src/wirings/ai-agent/ai-agent-agui.ts
  • packages/core/src/wirings/ai-agent/ai-agent-helpers.ts
  • packages/core/src/wirings/ai-agent/ai-agent-interrupt.test.ts
  • packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts
  • packages/core/src/wirings/ai-agent/ai-agent-prepare.ts
  • packages/core/src/wirings/ai-agent/ai-agent-runner.ts
  • packages/core/src/wirings/ai-agent/ai-agent-stream.test.ts
  • packages/core/src/wirings/ai-agent/ai-agent-stream.ts
  • packages/core/src/wirings/ai-agent/ai-agent.types.ts
  • packages/core/src/wirings/ai-agent/index.ts
  • packages/core/src/wirings/ai-agent/voice-input.test.ts
  • packages/core/src/wirings/ai-agent/voice-input.ts
  • packages/core/src/wirings/ai-agent/voice-output.test.ts
  • packages/core/src/wirings/ai-agent/voice-output.ts
  • packages/core/src/wirings/http/http.types.ts
  • packages/core/src/wirings/http/web-request.test.ts
  • packages/core/src/wirings/http/web-request.ts
  • packages/core/src/wirings/rpc/rpc-runner.ts
  • packages/core/src/wirings/rpc/rpc-types.ts
  • packages/services/ai-deepinfra/LICENSE
  • packages/services/ai-deepinfra/README.md
  • packages/services/ai-deepinfra/package.json
  • packages/services/ai-deepinfra/run-tests.sh
  • packages/services/ai-deepinfra/src/deepinfra-config.ts
  • packages/services/ai-deepinfra/src/index.ts
  • packages/services/ai-deepinfra/src/speech-model.test.ts
  • packages/services/ai-deepinfra/src/speech-model.ts
  • packages/services/ai-deepinfra/src/transcription-model.test.ts
  • packages/services/ai-deepinfra/src/transcription-model.ts
  • packages/services/ai-deepinfra/tsconfig.json
  • packages/services/ai-vercel/src/message-converter.test.ts
  • packages/services/ai-vercel/src/message-converter.ts
  • packages/services/ai-vercel/src/vercel-ai-agent-runner.test.ts
  • packages/services/ai-vercel/src/vercel-ai-agent-runner.ts
  • packages/services/kysely/src/kysely-tables.ts
  • packages/voice-agents/README.md
  • packages/voice-agents/package.json
  • packages/voice-agents/run-tests.sh
  • packages/voice-agents/src/audio-playback-queue.test.ts
  • packages/voice-agents/src/audio-playback-queue.ts
  • packages/voice-agents/src/index.ts
  • packages/voice-agents/src/silence-detector.ts
  • packages/voice-agents/src/speech-detector.ts
  • packages/voice-agents/src/spoken-approval.test.ts
  • packages/voice-agents/src/spoken-approval.ts
  • packages/voice-agents/src/use-voice-conversation.ts
  • packages/voice-agents/src/voice-session.ts
  • packages/voice-agents/tsconfig.cjs.json
  • packages/voice-agents/tsconfig.json
  • verifiers/middleware-and-permissions/src/start.ts

Comment on lines +9 to +13
aiMiddleware: [
voiceInput({
model: 'deepinfra/nvidia/Nemotron-3.5-ASR-Streaming-Multilingual-0.6b',
}),
],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Confirm that Nemotron is intended here.

voice-assistant.agent.ts records that Nemotron truncates the tail of short clips and that Whisper replaced it for that reason. voice-demo.function.ts also keeps Nemotron only as a comparison model. This agent now adopts Nemotron as its primary transcription model, which points the opposite way. If the choice is deliberate, add a short comment stating why this agent differs. If not, use deepinfra/openai/whisper-large-v3-turbo here too.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@e2e/packages/functions/src/agents/voice.agent.ts` around lines 9 - 13,
Resolve the model inconsistency in the voice agent’s aiMiddleware configuration:
use deepinfra/openai/whisper-large-v3-turbo as the primary voiceInput model
unless Nemotron is intentionally required here. If retaining Nemotron, add a
brief comment explaining why voice.agent differs from voice-assistant.agent and
voice-demo.function.

Comment on lines +44 to +93
const distOf = (specifier: string) =>
dirname(fileURLToPath(import.meta.resolve(specifier)))

/**
* Files the page may load, and which package each comes from.
*
* An allow-list rather than path sanitising: this route reads from disk under a
* name that arrives in a URL, and the set of files a demo needs is small,
* known, and never user-supplied.
*
* The first group is the voice package itself, imported as modules —
* `voice-session.js` pulls in the two detectors by relative import, so they
* have to be here too. The rest exist only for the Silero turn detector, which
* is opt-in on the page and served locally rather than from the library's CDN
* so it can be tried offline and so the ~2MB model download is visible in the
* network tab where it belongs.
*
* `@ricky0123/vad-web`'s browser bundle is UMD expecting a global `ort`, which
* is why onnxruntime's classic-script build is here rather than its ESM one.
* Its `.mjs`/`.wasm` pairs are fetched by onnxruntime itself at runtime, not by
* the page — which variant depends on what the browser supports, so all of
* them are listed.
*/
const SERVABLE = new Map<string, string>([
...(
[
'voice-session.js',
'silence-detector.js',
'speech-detector.js',
'audio-playback-queue.js',
'spoken-approval.js',
] as const
).map((file) => [file, '@pikku/voice-agents'] as [string, string]),
...(
[
'bundle.min.js',
'vad.worklet.bundle.min.js',
'silero_vad_v5.onnx',
] as const
).map((file) => [file, '@ricky0123/vad-web'] as [string, string]),
...(
[
'ort.wasm.min.js',
'ort-wasm-simd-threaded.mjs',
'ort-wasm-simd-threaded.wasm',
'ort-wasm-simd-threaded.jsep.mjs',
'ort-wasm-simd-threaded.jsep.wasm',
] as const
).map((file) => [file, 'onnxruntime-web'] as [string, string]),
])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Locate the resolved entry points and the requested asset files for each package.
set -euo pipefail

for pkg in "`@ricky0123/vad-web`" "onnxruntime-web" "`@pikku/voice-agents`"; do
  echo "=== $pkg ==="
  fd -H -t f 'package.json' node_modules/"$pkg" --max-depth 1 --exec jq '{name, main, module, exports}' {} \; 2>/dev/null || echo "not installed in sandbox"
done

echo "=== asset locations ==="
for f in bundle.min.js vad.worklet.bundle.min.js silero_vad_v5.onnx ort.wasm.min.js ort-wasm-simd-threaded.mjs; do
  fd -H -t f "^${f}$" node_modules 2>/dev/null | head -5
done

Repository: pikkujs/pikku

Length of output: 329


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== locate voice-demo function ==="
fd -H -t f 'voice-demo\.function\.ts$' . || true

file="$(fd -H -t f 'voice-demo\.function\.ts$' . | head -1)"
if [ -n "${file:-}" ]; then
  echo "=== outline ==="
  ast-grep outline "$file" --view expanded || true
  echo "=== lines 1-140 ==="
  sed -n '1,140p' "$file" | nl -ba
fi

echo "=== package lock excerpts ==="
if [ -f yarn.lock ]; then
  rg -n '"(`@ricky0123/vad-web`"|"onnxruntime-web"|"`@pikku/voice-agents`"|`@ricky0123/vad-web`|onnxruntime-web|`@pikku/voice-agents`)' yarn.lock | head -80
fi
for f in pnpm-lock.yaml package-lock.json yarn.lock; do
  [ -f "$f" ] && echo "--- $f package refs ---" && rg -n -F '`@ricky0123/vad-web`' -F 'onnxruntime-web' -F '`@pikku/voice-agents`' "$f" | head -80 || true
done

Repository: pikkujs/pikku

Length of output: 739


🌐 Web query:

@ricky0123/vad-web package.json main exports bundle.min.js vad.worklet.bundle.min.js silero_vad_v5.onnx onnxruntime-web package.json exports ort.wasm.min.js

💡 Result:

The @ricky0123/vad-web package is a client-side voice activity detection (VAD) library for the browser that relies on ONNX Runtime Web for model execution [1][2]. Key file and asset details for integration include: 1. Bundle Files: The primary library entry point for browser script tags is dist/bundle.min.js [3][4]. For its internal processing, the library requires a Web Audio API worklet, located at dist/vad.worklet.bundle.min.js [2][5]. 2. Model Files: The package includes Silero VAD models in ONNX format, specifically silero_vad_v5.onnx and a legacy version, silero_vad_legacy.onnx [2][5]. These must be accessible to the library at runtime [2]. 3. ONNX Runtime Web: The library requires onnxruntime-web as a dependency [1][2]. When using script tags, you must load the ONNX Runtime Web entry point, typically ort.js or similar, and ensure the WebAssembly binary (ort.wasm.min.js) is available for the library to load [1][4][2]. 4. Configuration: When initializing the library, you must provide base paths so it can locate these assets [2][5]: - baseAssetPath: The directory where dist/vad.worklet.bundle.min.js, silero_vad_v5.onnx, and related assets are hosted [2][5]. - onnxWASMBasePath: The directory where the onnxruntime-web WASM files (including ort.wasm.min.js) are hosted [2][5]. If you are using a bundler (like Vite or Webpack), you will typically need to copy these specific files from the node_modules/@ricky0123/vad-web/dist/ and node_modules/onnxruntime-web/dist/ directories into your public or output folder so they are accessible at the specified paths at runtime [2].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="$(fd -H -t f 'voice-demo\.function\.ts$' . | head -1)"
echo "=== file ==="
printf '%s\n' "$file"
echo "=== lines 1-140 ==="
sed -n '1,140p' "$file" | awk '{printf "%4d\t%s\n", NR, $0}'

echo "=== package manager files ==="
git ls-files | egrep '(^|/)(pnpm-lock\.yaml|package-lock\.json|yarn\.lock|package\.json)$' | head -50

Repository: pikkujs/pikku

Length of output: 8571


🌐 Web query:

registry.npmjs.org @ricky0123/vad-web 1.5.4 package.json exports

💡 Result:

As of August 2, 2026, the package @ricky0123/vad-web does not have a version 1.5.4; the latest available version is 0.0.30 [1][2][3]. Furthermore, the @ricky0123/vad-web package does not define an exports field in its package.json [4][5][6]. Instead of using the exports field, the package exposes its functionality primarily through the main entry point and direct imports from the dist/ directory [4][5][6]. Common patterns for importing from this package include: import { MicVAD } from "@ricky0123/vad-web"; // Or, for specific components: import { MicVAD } from "@ricky0123/vad-web/dist/index"; [7] The library's structure relies on a flat set of files within the dist/ folder, including index.js, real-time-vad.js, and various worker/model bundles [4]. Developers are typically advised to reference these files directly or import from the root package path, and static assets (like.onnx model files and.wasm files) must often be manually managed and served by the application's build system (e.g., Vite or Webpack) [8][9][7].

Citations:


🌐 Web query:

registry.npmjs.org onnxruntime-web package.json exports default import main "ort.wasm.min.js" "ort-wasm-simd-threaded.mjs"

💡 Result:

In the onnxruntime-web package, the package.json file uses the exports field to define conditional entry points for various environments and bundling strategies [1]. When importing onnxruntime-web in a JavaScript environment, the specific file loaded depends on your import style (ESM vs. CommonJS) and the target platform [1]. The files you mentioned, ort.wasm.min.js, ort-wasm-simd-threaded.mjs, and ort-wasm-simd-threaded.wasm, are core components of the WebAssembly execution provider [2][3]. Key details: 1. Entry Points and Imports: - When using standard imports (e.g., import * as ort from 'onnxruntime-web'), the package resolves to dist/ort.bundle.min.mjs (for ESM) or dist/ort.min.js (for CommonJS) by default [1]. - You can specifically target the WebAssembly-only runtime by importing from onnxruntime-web/wasm [1]. 2. Role of WASM Files: - ort-wasm-simd-threaded.mjs and ort-wasm-simd-threaded.wasm are the default WebAssembly backend artifacts [3]. As of version 1.19.0, multi-threaded SIMD is the standard and only supported WASM build, replacing previous non-threaded variants [3]. - ort.wasm.min.js acts as the JavaScript entry point for the WASM backend, often used in scenarios like Web Workers where scripts need to be loaded via importScripts() or direct script tags [4][5]. 3. Usage and Configuration: - These files are typically located in the dist/ directory of the package [1][4]. - If you are hosting the files yourself (rather than using a CDN), you must ensure that the onnxruntime-web library can locate these artifacts at runtime [6][5]. You can configure this using ort.env.wasm.wasmPaths to point to the directory where you have deployed the .wasm and .mjs files [5][7]. - For applications using bundlers like Vite or Webpack, the bundler often handles these assets automatically; however, if you are working with Web Workers or custom setups, you may need to manually copy these files to your public static assets folder to ensure they are accessible at the correct URL paths [4][5]. In summary, ort-wasm-simd-threaded.mjs and its accompanying .wasm file provide the high-performance inference engine, while ort.wasm.min.js serves as the loader/entry point for the WASM provider [4][3]. Direct interaction with these files is generally handled by the library itself, provided they are correctly discoverable in your environment [5][7].

Citations:


🌐 Web query:

registry.npmjs.org onnxruntime-web 1.19.0 package.json exports import default types

💡 Result:

In onnxruntime-web version 1.19.0, the package.json file utilizes the "exports" field to manage conditional imports, allowing developers to import specific subsets of the library based on their execution provider or environment needs [1][2]. The types for the package are provided via the "types" property within each entry of the "exports" field, pointing to the root "types.d.ts" file [1]. When importing the package, you can use either ES Modules (recommended) or CommonJS syntax [3][4]. The "exports" field defines the following mappings [1][2]: 1. Default Import (onnxruntime-web): Accessible via import * as ort from 'onnxruntime-web' (ESM) or const ort = require('onnxruntime-web') (CJS) [3][2][4]. 2. Conditional Sub-paths: You can reduce bundle sizes by importing only necessary features, such as: - WebGPU: onnxruntime-web/webgpu [3][4] - WebGL: onnxruntime-web/webgl [4] - Wasm: onnxruntime-web/wasm [4] - Experimental: onnxruntime-web/experimental [3][4] - Training: onnxruntime-web/training [4] The package structure within "exports" explicitly separates "import" (for ESM) and "require" (for CJS) conditions, and often provides bundles for different environments (e.g., node, web) [1]. Since version 1.19, the library optimized its module export to better support these modular imports [2]. If you are using TypeScript, the types are automatically resolved through the "types" definition in the "exports" mapping [1].

Citations:


🌐 Web query:

registry.npmjs.org onnxruntime-web 1.19.0 package.json

💡 Result:

The onnxruntime-web version 1.19.0 package.json file, hosted on registry.npmjs.org, defines the library's metadata and structure for the ONNX Runtime Web v1.19.0 release (published August 2024) [1]. The package.json specifies the entry points and exports for the library, facilitating its use across Node.js and browser environments [2]. Key features of the package.json structure for this version include: Main and Browser Entry Points: - main: "dist/ort.node.min.js" - browser: "dist/ort.min.js" Exports Mapping: The "exports" field provides conditional exports to handle different module systems (ESM and CJS) and environments: - Root ("."): Configured with specific paths for node (import/require), and ESM import configurations including "onnxruntime-web-use-extern-wasm" and a default bundle ("dist/ort.bundle.min.mjs") [2]. - Additional entry points: "./all", "./wasm", "./webgl", and "./webgpu" are also defined to allow selective importing of components, such as specific execution providers [2]. Release Context: Version 1.19.0 introduced updates to JavaScript packaging to align with modern best practices, which notably included slight incompatibilities for applications bundling the library [1]. This release also necessitated specific runtime files (e.g.,.mjs files in the /dist directory) that caused integration issues for some CDNs like cdnjs that had not yet indexed the full file set [3]. You can view the raw content of the package.json file for this specific version directly via UNPKG at: https://app.unpkg.com/onnxruntime-web@1.19.0/files/package.json [4].

Citations:


🌐 Web query:

registry.npmjs.org @ricky0123/vad-web 0.0.30 package.json

💡 Result:

The package @ricky0123/vad-web version 0.0.30 was published to the npm registry on November 21, 2025 [1][2][3]. It is a voice activity detection library designed for browser environments, utilizing onnxruntime-web [1][4][5]. While the full JSON contents are managed by the npm registry, the package structure at version 0.0.30 includes a dist directory containing the bundle (bundle.min.js, bundle.dev.js), various TypeScript definition files (.d.ts), and model-related JavaScript files [6][7]. You can access the package.json file specifically for version 0.0.30 directly through the npm registry's endpoint or via community tools that index package source code [6]. The package is licensed under the ISC license and depends on onnxruntime-web [1][2].

Citations:


🌐 Web query:

cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.30/package.json

💡 Result:

The file located at cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.30/package.json is the package manifest for version 0.0.30 of the @ricky0123/vad-web npm package [1][2]. This package provides a voice activity detector (VAD) designed for browser environments [3][4][5]. Version 0.0.30 was published on November 21, 2025 [4]. The package.json file defines metadata for the project, including its dependencies (such as onnxruntime-web), entry points, and project configuration [3][4]. You can view the contents of this file directly through various CDN and package registry interfaces that host the package [1][6].

Citations:


🌐 Web query:

cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.30/package.json main module exports

💡 Result:

The @ricky0123/vad-web package, specifically version 0.0.30, is designed for browser-based voice activity detection [1][2]. Based on the provided file structure and documentation, the package provides its functionality via modules located in the dist directory [2][3]. While the specific raw content of the package.json file was not extracted, the documentation and project file structures confirm that the package is an ESM-compatible library [4]. In typical modern usage, the package provides named exports such as MicVAD, which users import directly from the package [5][6][7]. Developers generally access the library by importing from the package name, for example: import { MicVAD } from "@ricky0123/vad-web"; Internally, the package maps these exports to files within the dist folder, such as dist/index.js [2][3]. Because the package relies on browser-specific APIs and assets like Web Audio worklets and ONNX runtime files, bundling often requires configuring tools like Vite or Webpack to correctly serve the static files located in the dist directory (e.g., vad.worklet.bundle.min.js, silero_vad_v5.onnx, and WASM binaries) [8][9][10][7].

Citations:


🌐 Web query:

"https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.30/package.json" | sed -n '1,80p'

💡 Result:

The package @ricky0123/vad-web version 0.0.30 is a JavaScript library designed for performing Voice Activity Detection (VAD) directly in the web browser [1][2]. It allows developers to easily detect when a user is speaking by leveraging machine learning models (specifically using onnxruntime-web) [3][4]. The package is part of the ricky0123/vad project, which focuses on providing accurate, user-friendly voice activity detection [4][2]. Version 0.0.30 was published on November 21, 2025 [5][6]. Key updates in this release included fixes for custom audio context handling, a memory leak related to onnxruntime, and various API changes [6]. You can install it via npm or yarn: npm install @ricky0123/vad-web@0.0.30 [7] Usage typically involves importing MicVAD and initializing it to process microphone input [8]: import { MicVAD } from "@ricky0123/vad-web" const myvad = await MicVAD.new({ onSpeechEnd: (audio) => { // Process audio (Float32Array) }, }) myvad.start Documentation and further examples can be found at https://docs.vad.ricky0123.com/ [4].

Citations:


🌐 Web query:

npmx.dev package-code @ricky0123/vad-web v0.0.30 package.json

💡 Result:

The package.json file for @ricky0123/vad-web version 0.0.30 can be accessed and viewed online via the npmx.dev service at the following URL: https://npmx.dev/package-code/@ricky0123/vad-web/v/0.0.30/package.json [1]. This file is also available on other package content delivery networks such as UNPKG: https://app.unpkg.com/@ricky0123/vad-web@0.0.30/files/package.json [2]. The @ricky0123/vad-web package, maintained by ricky0123, is a browser-based voice activity detector [3][4]. Version 0.0.30 was published on November 21, 2025 [5][6].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== raw vad package manifests ==="
for version in 0.0.30; do
  pkg='`@ricky0123/vad-web`'
  echo "--- $pkg@$version package.json ---"
  curl -fsSL "https://cdn.jsdelivr.net/npm/$pkg@$version/package.json" | sed -n '1,80p' || true
done

echo "=== raw onnxruntime-web package manifests ==="
for version in 1.19.0 1.19.2; do
  pkg='onnxruntime-web'
  echo "--- $pkg@$version package.json ---"
  curl -fsSL "https://cdn.jsdelivr.net/npm/$pkg@$version/package.json" | sed -n '1,120p' || true
done

echo "=== resolve dist directories from manifests by package root ==="
python3 - <<'PY'
import json, urllib.request

packages = [("onnxruntime-web", "1.19.2"), ("`@ricky0123/vad-web`", "0.0.30")]
for p,v in packages:
    url = "https://cdn.jsdelivr.net/npm/"+p+"@"+v+"/package.json"
    pkg=j=json.load(urllib.request.urlopen(url))
    print(f"{p}@{v} main={pkg.get('main')!r} module={pkg.get('module')!r} browser={pkg.get('browser')!r} exports={json.dumps(pkg.get('exports',''), sort_keys=True)[:200]!r}")
PY

Repository: pikkujs/pikku

Length of output: 10911


Resolve assets from their real package asset directory.

distOf uses the package entry point directory, but onnxruntime-web resolves ESM to dist/ort.bundle.min.mjs, and @ricky0123/vad-web resolves to dist/index.js. bundle.min.js is at the package root, while the listed onnxruntime-web assets are at dist, so /voice-demo/lib/bundle.min.js can return 404 instead of a clear failure. Use package-specific resolve targets for each allow-list group.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@e2e/packages/functions/src/functions/voice-demo.function.ts` around lines 44
- 93, Update the asset resolution used with SERVABLE and distOf so each package
group resolves from the correct asset directory rather than assuming the package
entry-point directory. Use package-specific resolve targets: the voice assets
from the voice package, vad-web’s root-level bundle assets from its package
root, and onnxruntime-web’s listed runtime assets from its dist directory;
preserve the existing allow-list and clear failure behavior for unresolved
assets.

Comment on lines +304 to +318
/**
* Stop the agent talking.
*
* This is what `agentInterrupt()` wraps, written out because the inspector
* reads `func` statically and cannot see through a spread. `rpc.agent.interrupt`
* checks the run belongs to the caller before stopping it, and resolves
* `stopped: false` when there was nothing left to stop — racing a run that
* finishes on its own is the normal case here, not an error.
*/
export const voiceDemoInterrupt = pikkuSessionlessFunc({
description: 'Interrupts the run the demo page is currently listening to',
input: VoiceDemoInterruptInput,
func: async (_services, { runId, reason }, { rpc }) =>
rpc.agent.interrupt(runId, reason),
})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Inspect the agent interrupt RPC to see how it establishes run ownership.
set -euo pipefail

fd -t f 'ai-agent-interrupt.ts' packages -x sed -n '1,200p' {}

echo "=== interrupt registrations / ownership checks ==="
rg -nP -C 6 '\binterrupt\s*[:(]' packages/core/src --type=ts -g '!**/*.test.ts'

echo "=== session/userId use in interrupt paths ==="
rg -nP -C 4 '(userId|session|resourceId)' packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts 2>/dev/null || true

Repository: pikkujs/pikku

Length of output: 11655


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== ai-agent-interrupt implementation around ownership comments ==="
sed -n '180,260p' packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts

echo "=== rpc-runner interrupt implementation ==="
sed -n '500,545p' packages/core/src/wirings/rpc/rpc-runner.ts

echo "=== function wiring declaration ==="
fd -t f 'voice-demo.function.ts|voice-demo.wirings.ts' e2e -x sh -c 'echo "--- $1"; sed -n "280,330p" "$1"' sh {}

Repository: pikkujs/pikku

Length of output: 6388


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== interruptAIAgent implementation ==="
grep -n "export const interruptAIAgent\|interruptAIAgent" packages/core/src/wirings/ai-agent/*.ts
fd -t f 'ai-agent-interrupt.ts|ai-agent-*.ts' packages/core/src/wirings/ai-agent -x sh -c 'echo "--- $1"; rg -n -C 4 -C 12 "export const interruptAIIgent|function interruptAIIgent|const interruptAIIgent|interruptAIIgent" "$1"' sh {}
# exact spelling:
rg -n -C 8 "export const interruptAIAgent|function interruptAIAgent|const interruptAIAgent|interruptAIAgent" packages/core/src/wirings/ai-agent packages/core/src --type=ts

echo "=== voice-demo wiring full ==="
fd -t f 'voice-demo.wirings.ts' e2e -x sh -c 'echo "--- $1"; cat -n "$1"' sh {}

Repository: pikkujs/pikku

Length of output: 1919


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== interruptAIAgent implementation ==="
sed -n '940,1045p' packages/core/src/wirings/ai-agent/ai-agent-stream.ts

echo "=== interruptAIAgent authorization test ==="
sed -n '682,845p' packages/core/src/wirings/ai-agent/ai-agent-interrupt.test.ts

echo "=== voice-demo wiring ==="
fd -t f 'voice-demo.wirings.ts' e2e -x sh -c 'echo "--- $1"; cat -n "$1"' sh {}

Repository: pikkujs/pikku

Length of output: 10431


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== voice-demo function interrupt registration ==="
sed -n '1,180p' e2e/packages/functions/src/functions/voice-demo.function.ts
sed -n '300,450p' e2e/packages/functions/src/functions/voice-demo.function.ts

echo "=== session resolution helper ==="
rg -n -C 5 "export function |function resolveOwnerResourceId|resolveOwnerResourceId|assertResourceOwner" packages/core/src --type=ts

echo "=== all voice-demo run creation references ==="
rg -n -C 3 "runId|voiceDemo|interrupt" e2e/packages/functions/src/functions/voice-demo.function.ts e2e/packages/functions/src/wirings/voice-demo.wirings.ts

Repository: pikkujs/pikku

Length of output: 50372


No ownership check exists on this route.

rpc.agent.interrupt calls interruptAIAgent, which compares the session principal to the run ownership and rejects missing sessions. Because this is a pikkuSessionlessFunc wired with auth: false, the demo interrupt cannot rely on ownership proof; either this route should establish the demo’s owner/principal, accept only owned runs during stream runtime, or correct the comment to not assert an absent check.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@e2e/packages/functions/src/functions/voice-demo.function.ts` around lines 304
- 318, The voiceDemoInterrupt route currently claims rpc.agent.interrupt
performs caller ownership validation despite being sessionless with auth
disabled. Update the route to establish and pass the demo owner/principal before
calling rpc.agent.interrupt, or restrict interruption to runs verified as owned
during stream runtime; otherwise remove the inaccurate ownership assertion from
the function comment.

Comment on lines +748 to +755
const { stopped, inFlightTools } = await post('/voice-demo/interrupt', {
runId,
reason: 'speech',
})
if (stopped && inFlightTools.length > 0) {
// Named, not offered to undo — it has already happened.
say('note', 'still finishing: ' + inFlightTools.join(', '))
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Guard inFlightTools before reading length.

The && short-circuit protects inFlightTools only when stopped is falsy. If the interrupt response omits inFlightTools while stopped is true, inFlightTools.length throws. The surrounding catch then shows the TypeError as a page note, which reads as an interrupt failure even though the interrupt succeeded.

🛡️ Proposed fix
-    if (stopped && inFlightTools.length > 0) {
+    if (stopped && inFlightTools?.length > 0) {
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const { stopped, inFlightTools } = await post('/voice-demo/interrupt', {
runId,
reason: 'speech',
})
if (stopped && inFlightTools.length > 0) {
// Named, not offered to undo — it has already happened.
say('note', 'still finishing: ' + inFlightTools.join(', '))
}
const { stopped, inFlightTools } = await post('/voice-demo/interrupt', {
runId,
reason: 'speech',
})
if (stopped && inFlightTools?.length > 0) {
// Named, not offered to undo — it has already happened.
say('note', 'still finishing: ' + inFlightTools.join(', '))
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@e2e/packages/functions/src/functions/voice-demo.function.ts` around lines 748
- 755, Guard inFlightTools before accessing its length in the interrupt handling
logic after post('/voice-demo/interrupt'). Preserve the existing note behavior
when stopped is true and tools are present, while treating an omitted or
undefined inFlightTools value as an empty list.

Comment on lines +880 to +885
} catch (error) {
say('note', 'microphone unavailable: ' + error.message)
session = null
playback = null
$vad.disabled = false
setState('idle')

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Destroy the playback queue when the session fails to start.

startSession constructs AudioPlaybackQueue before it constructs VoiceSession. If the microphone prompt is denied, or session.start() throws for any other reason, this catch sets playback to null without calling destroy(). The AudioContext it holds is never closed. Browsers cap concurrent AudioContexts, so repeated denied attempts — the common case this branch exists for — eventually fail with an unrelated error.

🛡️ Proposed fix
   } catch (error) {
     say('note', 'microphone unavailable: ' + error.message)
+    void playback?.destroy()
+    void session?.destroy()
     session = null
     playback = null
     $vad.disabled = false
     setState('idle')
   } finally {
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
} catch (error) {
say('note', 'microphone unavailable: ' + error.message)
session = null
playback = null
$vad.disabled = false
setState('idle')
} catch (error) {
say('note', 'microphone unavailable: ' + error.message)
void playback?.destroy()
void session?.destroy()
session = null
playback = null
$vad.disabled = false
setState('idle')
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@e2e/packages/functions/src/functions/voice-demo.function.ts` around lines 880
- 885, Update the startSession error path around the shown catch block to
destroy the existing AudioPlaybackQueue before clearing playback. Invoke
playback.destroy() when the queue was created, then set playback to null while
preserving the existing session reset and idle-state behavior.

Comment on lines +140 to +153
// Silence comes from the empty queue, not from a suspended context. Barge-in
// reaches here via `pause()`, and leaving the context suspended would mean
// the *next* reply decodes, queues and starts a source node that plays to
// nobody — the conversation goes quiet from the second turn onwards.
void this.resume()

return { text: heard.join(' '), complete }
}

/** Release the audio context. The queue is unusable afterwards. */
async destroy(): Promise<void> {
this.interrupt()
await this.ctx.close()
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

destroy() can produce an unhandled promise rejection.

destroy() calls interrupt(), which starts this.resume() without awaiting it. destroy() then closes the context. If close() settles first, resume() rejects with InvalidStateError on a closed AudioContext. The void discards that rejection, so it surfaces as an unhandled rejection.

Attach a catch, or skip the resume when the context is closing.

🐛 Proposed fix
-    void this.resume()
+    void this.resume().catch(() => {
+      // The context was closed underneath us (destroy); nothing to resume.
+    })
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Silence comes from the empty queue, not from a suspended context. Barge-in
// reaches here via `pause()`, and leaving the context suspended would mean
// the *next* reply decodes, queues and starts a source node that plays to
// nobody — the conversation goes quiet from the second turn onwards.
void this.resume()
return { text: heard.join(' '), complete }
}
/** Release the audio context. The queue is unusable afterwards. */
async destroy(): Promise<void> {
this.interrupt()
await this.ctx.close()
}
// Silence comes from the empty queue, not from a suspended context. Barge-in
// reaches here via `pause()`, and leaving the context suspended would mean
// the *next* reply decodes, queues and starts a source node that plays to
// nobody — the conversation goes quiet from the second turn onwards.
void this.resume().catch(() => {
// The context was closed underneath us (destroy); nothing to resume.
})
return { text: heard.join(' '), complete }
}
/** Release the audio context. The queue is unusable afterwards. */
async destroy(): Promise<void> {
this.interrupt()
await this.ctx.close()
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/src/audio-playback-queue.ts` around lines 140 - 153,
Update destroy() and the interrupt/resume flow so the asynchronous resume
triggered by interrupt() cannot become an unhandled rejection when ctx.close()
runs; either await and handle the resume before closing the AudioContext or
attach an explicit rejection handler, while preserving the queue’s existing
interruption behavior.

Comment on lines +262 to +277
try {
await vad.start()
} catch (error) {
handlers.onError?.(
error instanceof Error ? error : new Error(String(error))
)
}

return {
stop: () => {
if (stopped) return
stopped = true
void vad.destroy()
},
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Catch vad.destroy() rejections.

stop() calls void vad.destroy() without a .catch(). If destroy() rejects — for example after a vad.start() failure at line 264 left the instance partially initialized — this becomes an unhandled promise rejection.

🔧 Proposed fix
   return {
     stop: () => {
       if (stopped) return
       stopped = true
-      void vad.destroy()
+      vad.destroy().catch(() => {})
     },
   }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
try {
await vad.start()
} catch (error) {
handlers.onError?.(
error instanceof Error ? error : new Error(String(error))
)
}
return {
stop: () => {
if (stopped) return
stopped = true
void vad.destroy()
},
}
}
try {
await vad.start()
} catch (error) {
handlers.onError?.(
error instanceof Error ? error : new Error(String(error))
)
}
return {
stop: () => {
if (stopped) return
stopped = true
vad.destroy().catch(() => {})
},
}
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/src/speech-detector.ts` around lines 262 - 277, Update
the stop method returned by the speech detector setup to attach a rejection
handler to vad.destroy(). Route any destruction error through handlers.onError,
preserving the existing stopped guard and fire-and-forget behavior.

Comment on lines +131 to +147
const DENIALS = [
'do not',
'dont',
'never mind',
'nevermind',
'hold on',
'wait',
'stop',
'cancel',
'forget it',
'negative',
'denied',
'deny',
'no',
'nope',
'nah',
]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

interpretConsent reads "not okay" as consent.

normalise keeps the word "not". DENIALS has no bare not entry, and \bno\b does not match "not". So takePhrases finds the grant word and no denial. interpretConsent('not okay') returns 'granted'. The same applies to "not yes", "not fine", and "that's not okay". This grants a tool approval that the user refused.

Detect a negator before a grant phrase, or add not to the denial phrases.

🐛 Proposed fix
 const DENIALS = [
   'do not',
   'dont',
+  'not',
   'never mind',

Add covering cases to spoken-approval.test.ts:

test('does not read a negated grant as consent', () => {
  assert.equal(interpretConsent('not okay'), 'denied')
  assert.equal(interpretConsent('that is not fine'), 'denied')
})

Also applies to: 181-192

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/src/spoken-approval.ts` around lines 131 - 147, Update
interpretConsent and its phrase matching around DENIALS/takePhrases so negated
grant phrases such as “not okay” and “that is not fine” return “denied” rather
than “granted”; detect “not” before a grant phrase or include it in the denial
handling, and add the specified covering assertions to spoken-approval.test.ts.

Comment on lines +21 to +24
/** Milliseconds of silence that end a turn. Defaults to 1200. */
silenceDelay?: number
/** Analyser noise floor in dBFS. Defaults to -50. */
minDecibels?: number

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Fix the documented default for silenceDelay.

The doc comment states 1200. captureNextTurn uses this.options.silenceDelay ?? 700 at Line 322. Align the doc with the code.

📝 Proposed doc fix
-  /** Milliseconds of silence that end a turn. Defaults to 1200. */
+  /** Milliseconds of silence that end a turn. Defaults to 700. */
   silenceDelay?: number
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
/** Milliseconds of silence that end a turn. Defaults to 1200. */
silenceDelay?: number
/** Analyser noise floor in dBFS. Defaults to -50. */
minDecibels?: number
/** Milliseconds of silence that end a turn. Defaults to 700. */
silenceDelay?: number
/** Analyser noise floor in dBFS. Defaults to -50. */
minDecibels?: number
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/src/voice-session.ts` around lines 21 - 24, Update the
silenceDelay documentation in the voice session options to state the actual
default of 700 milliseconds, matching the fallback used by captureNextTurn;
leave the minDecibels documentation unchanged.

Comment on lines +134 to +150
async start(): Promise<void> {
if (this.listening) return
await this.acquire()
// Before `listening`, so the first turn is armed knowing which detector it
// has. Loading the model takes a second or two on a cold cache, and that
// wait belongs here rather than in the middle of the user's first sentence.
if (this.options.speech) await this.startSpeechDetector()
this.listening = true
this.captureNextTurn()
}

/** Stop waiting for turns but keep the microphone open, so resuming does not
* renegotiate the device. Use this between conversations, not {@link destroy}. */
stop(): void {
this.listening = false
this.endTurn()
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

start() after stop() creates a second speech detector and leaks the first.

stop() clears listening and calls endTurn(), but it never calls this.stopSpeech. The detector stays armed. A later start() runs startSpeechDetector() again, because acquire() returns early and nothing guards on this.stopSpeech. The new stop function overwrites this.stopSpeech, so the first MicVAD instance can never be destroyed. It keeps running the model and keeps invoking onSpeechEnd, so each turn is emitted twice once listening is true again.

useVoiceConversation reaches this path directly: stop() then start() reuses the existing sessionRef.current and calls session.start().

Guard the detector startup so it is armed at most once per acquired stream.

🐛 Proposed fix
   async start(): Promise<void> {
     if (this.listening) return
     await this.acquire()
     // Before `listening`, so the first turn is armed knowing which detector it
     // has. Loading the model takes a second or two on a cold cache, and that
     // wait belongs here rather than in the middle of the user's first sentence.
-    if (this.options.speech) await this.startSpeechDetector()
+    if (this.options.speech && !this.stopSpeech) {
+      await this.startSpeechDetector()
+    }
     this.listening = true
     this.captureNextTurn()
   }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
async start(): Promise<void> {
if (this.listening) return
await this.acquire()
// Before `listening`, so the first turn is armed knowing which detector it
// has. Loading the model takes a second or two on a cold cache, and that
// wait belongs here rather than in the middle of the user's first sentence.
if (this.options.speech) await this.startSpeechDetector()
this.listening = true
this.captureNextTurn()
}
/** Stop waiting for turns but keep the microphone open, so resuming does not
* renegotiate the device. Use this between conversations, not {@link destroy}. */
stop(): void {
this.listening = false
this.endTurn()
}
async start(): Promise<void> {
if (this.listening) return
await this.acquire()
// Before `listening`, so the first turn is armed knowing which detector it
// has. Loading the model takes a second or two on a cold cache, and that
// wait belongs here rather than in the middle of the user's first sentence.
if (this.options.speech && !this.stopSpeech) {
await this.startSpeechDetector()
}
this.listening = true
this.captureNextTurn()
}
/** Stop waiting for turns but keep the microphone open, so resuming does not
* renegotiate the device. Use this between conversations, not {`@link` destroy}. */
stop(): void {
this.listening = false
this.endTurn()
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/voice-agents/src/voice-session.ts` around lines 134 - 150, Ensure
the start() flow arms the speech detector at most once per acquired stream by
guarding startSpeechDetector() with the existing detector state, such as
this.stopSpeech. Preserve detector reuse across stop() followed by start(),
while still starting it when no detector has been created.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant