Repository navigation
Voice agents: cooperative interrupt, @pikku/voice-agents, and store-backed converse assertions - #1085
Conversation
…ware Adds the primitive a verbal agent needs but a text one never did: the ability to stop mid-turn without losing what already happened. `abortScope` carries a cancellation signal down through the function runner, and `beginChanges()` gives an agent a checkpoint it can be interrupted at safely rather than anywhere. `interrupted` joins the AI run status enum, kept distinct from `failed`: the run was stopped on purpose, usually because the user talked over it, so it should not surface in error reporting. Also teaches `actor.converse` to say the word 'json' in its instructions. Every call it makes wants a schema'd object back, and a gateway that cannot take a JSON schema degrades to OpenAI's `json_object` mode, which refuses the request outright unless the prompt mentions json. Saying it once covers all three call sites. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DeepInfra fronts the STT/TTS models the voice path moved onto once Whisper's invented filler turned out not to be filterable by confidence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The browser half of a verbal conversation — mic capture, speech and silence detection, playback queueing and spoken approvals — kept as its own package so the API half can stay in core and neither has to know about the other's runtime. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…chemas Two defects that meant `actor.converse` had never once worked under `pikku scenario run`. Scenario steps execute in the CLI process, which stubbed its singleton services to three entries — so every conversing scenario died on AIProviderNotConfigured before saying anything. It now builds a runner the same way `pikku dev` does, and only when the project declares agents, so agent-less projects still need no AI env. The second is worse because it is silent: `supportsStructuredOutputs` defaults to false, which drops `outputSchema`, degrades the call to bare `json_object`, and returns well-formed JSON with whatever keys the model felt like. Nothing errors; the caller reads undefined off every field. Adds `scenarios.actorModel` so the actor's model is configurable — it is deliberately not the model under test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ence A voice agent once replied 'Okay — marking get lunch done.' having called no tool at all, and on another run reached for deleteTodo. Both pass a test that only reads the actor's verdict, because the actor is judging the conversation and the conversation was fine. So the assertions go to the store instead: the todo exists, it is completed, exactly one of it exists, and the todo nobody mentioned is untouched. Verified negatively — with completeTodo removed from the agent, the actor still returns passed:true and the store assertion is what fails. Note what is not asserted: which tools ran. The user asked for a todo to be done, not for completeTodo to be invoked. Whether the agent's sentence was honest about how it got there is a grade on the run rather than a check on the world, and belongs to the judge in #719. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
📝 WalkthroughWalkthroughChangesThe PR adds a voice-agent package, DeepInfra speech services, interruptible AI-agent runs, voice streaming support, browser voice demos, scenario integration, and binary HTTP response handling. Voice agent platform
AI runtime and provider services
CLI and voice-agent integration
Transport and middleware validation
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Browser
participant VoiceSession
participant voiceDemoRoutes
participant AIStream
participant AudioPlaybackQueue
Browser->>VoiceSession: capture microphone turn
VoiceSession->>voiceDemoRoutes: submit transcription or conversation input
voiceDemoRoutes->>AIStream: start or interrupt agent run
AIStream->>Browser: emit text, audio, or interruption events
Browser->>AudioPlaybackQueue: enqueue audio chunks
AudioPlaybackQueue->>Browser: report playback completion or heard text
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 20
🧹 Nitpick comments (23)
packages/core/src/wirings/http/web-request.test.ts (1)
354-369: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winCover the
sendbranch for binary bodies.
createMockResponsedoes not definesend. This test only validatesres.arrayBuffer(body).applyWebResponsecallsres.send(body)when an adapter provides it. Add asend-capable mock and assert the same byte sequence for that path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/core/src/wirings/http/web-request.test.ts` around lines 354 - 369, Extend the binary-body test around applyWebResponse to use a createMockResponse variant that defines send, then assert send receives the original byte sequence unchanged. Preserve the existing arrayBuffer assertion and verify the adapter-provided send branch with the same binary payload.e2e/packages/functions/src/functions/voice-demo.function.ts (2)
463-468: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueEncoding byte by byte adds latency to every turn.
toBase64concatenates one character per byte on the main thread.handleTurnawaits it before the transcription request starts, so the cost lands in the gap the page is trying to keep short. Chunked conversion removes the per-byte string growth.♻️ Proposed change
const toBase64 = async (blob) => { const bytes = new Uint8Array(await blob.arrayBuffer()) let binary = '' - for (let i = 0; i < bytes.length; i++) binary += String.fromCharCode(bytes[i]) + const CHUNK = 0x8000 + for (let i = 0; i < bytes.length; i += CHUNK) { + binary += String.fromCharCode.apply(null, bytes.subarray(i, i + CHUNK)) + } return btoa(binary) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@e2e/packages/functions/src/functions/voice-demo.function.ts` around lines 463 - 468, Update toBase64 to avoid byte-by-byte string concatenation: convert the Uint8Array to binary in chunks before calling btoa, preserving the same Base64 output and async behavior used by handleTurn.
144-148: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winBound the audio and text inputs.
audioandtextaccept a string of any length. Both routes are unauthenticated and both forward the payload to a billed provider. A single large request turns into a large provider charge, andBuffer.fromon a very large base64 string allocates the whole decoded buffer in memory. Add a.max()to each schema. A few seconds of Opus is a few tens of kilobytes, so the bound can be generous and still useful.♻️ Proposed change
export const VoiceDemoTranscribeInput = z.object({ /** The recorded turn, base64. JSON rather than a raw body so the route needs * no binary handling on either side; a few seconds of Opus is small. */ - audio: z.string(), + audio: z.string().max(4_000_000), })export const VoiceDemoSpeakInput = z.object({ - text: z.string(), + text: z.string().max(2_000), })Also applies to: 248-250
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@e2e/packages/functions/src/functions/voice-demo.function.ts` around lines 144 - 148, Bound the unauthenticated provider inputs by adding generous maximum-length validation to the audio field in VoiceDemoTranscribeInput and the text field in the corresponding text-input schema. Apply the limits to the z.string() definitions before payloads reach Buffer.from or the billed provider, preserving the existing schema behavior for valid short inputs.e2e/packages/functions/src/agents/voice-assistant.agent.ts (1)
120-150: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winShare the model ids and voice map with the demo route.
e2e/packages/functions/src/functions/voice-demo.function.tsredeclares the same values asASR_MODEL,TTS_MODEL, andSPEAKABLE_SCRIPTS. Its comment states that these must stay in step with this agent. A copy in two files drifts silently, and drift here changes the voice the user hears or the script coverage check. Export the constants from one module and import them in both places.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@e2e/packages/functions/src/agents/voice-assistant.agent.ts` around lines 120 - 150, Centralize the voice configuration currently passed to voiceInput and voiceOutput in this agent, including the ASR model id, TTS model id, and speakableScripts map, as exported constants. Update both the voice assistant agent and voice-demo.function.ts to import and use those shared constants, removing the duplicate declarations while preserving the existing model and voice values.packages/voice-agents/README.md (1)
51-58: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd the missing exports to the reference table.
The table lists four exports, but the README itself demonstrates
spokenApprovalandinterpretConsentimported from@pikku/voice-agents(line 113), anddetectSpeechis documented at length elsewhere in the package as an alternative todetectSilence. None of these three appear in the table. Add rows for them so the table remains a complete reference.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/voice-agents/README.md` around lines 51 - 58, Update the “What each piece is for” export table to include spokenApproval, interpretConsent, and detectSpeech, with concise purposes matching their documented behavior and the existing README terminology.packages/voice-agents/run-tests.sh (1)
4-4: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueHarden the shell script per shellcheck hints.
Shellcheck flags two spots:
- Line 4:
cd "$script_dir"has no failure guard (SC2164). Ifcdfails, the rest of the script runs from the wrong directory.- Line 35:
files=($(find src -type f -name "*.test.ts"))splits on whitespace (SC2207). A test file path containing a space breaks into multiple array elements.Both fixes are small and improve robustness.
🛠️ Proposed fixes
-cd "$script_dir" +cd "$script_dir" || exit 1-files=($(find src -type f -name "*.test.ts")) +mapfile -t files < <(find src -type f -name "*.test.ts")Also applies to: 35-35
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/voice-agents/run-tests.sh` at line 4, Harden the run-tests.sh setup and file collection: update the cd command near script_dir to fail immediately if changing directories fails, and replace the unquoted find-based array assignment near files with whitespace-safe path handling so filenames containing spaces remain single elements.Source: Linters/SAST tools
packages/voice-agents/src/speech-detector.ts (1)
156-277: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftAdd test coverage for the two turn-detection modules. Both
detectSpeechanddetectSilenceimplement timing- and callback-sensitive logic (redemption windows, misfire/short-segment discarding, idempotentstop(), progress throttling) with no accompanying test file in this batch, while sibling modules in the same layer (audio-playback-queue.ts,spoken-approval.ts) each have one.
packages/voice-agents/src/speech-detector.ts#L156-L277: addspeech-detector.test.tscoveringonSpeechStart/onSpeechEnd/onVADMisfirehandling, theredemptionMstiming adjustment inendTurn, and idempotentstop(), using a mockedVadModule/VadInstance.packages/voice-agents/src/silence-detector.ts#L44-L118: addsilence-detector.test.tscovering the audible/silent transition,onSilenceProgressthrottling viaPROGRESS_STEP, and idempotentstop(), using a mockedAudioContext/AudioNode/requestAnimationFrame.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/voice-agents/src/speech-detector.ts` around lines 156 - 277, Add speech-detector.test.ts covering detectSpeech callbacks for speech starts, speech ends, misfires, redemptionMs-adjusted timing, and idempotent stop() using mocked VadModule/VadInstance; add silence-detector.test.ts covering detectSilence audible-to-silent transitions, PROGRESS_STEP progress throttling, and idempotent stop() using mocked AudioContext, AudioNode, and requestAnimationFrame. Update only the tests; no direct production-code change is required in either detector.packages/voice-agents/package.json (1)
10-15: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd a
typesexport condition for@pikku/voice-agents.
packages/voice-agents/package.jsonexposesimportandrequireentrypoints, but consumers usingexportsneed an explicittypesentrypoint to resolve the emitted declarations. Add a top-level"types"field and list"types"first under"."inexports.📦 Proposed fix for the export map
+ "types": "./dist/esm/index.d.ts", "exports": { ".": { + "types": "./dist/esm/index.d.ts", "import": "./dist/esm/index.js", "require": "./dist/cjs/index.js" } },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/voice-agents/package.json` around lines 10 - 15, Add the package-level types field pointing to the emitted declarations, and update the "." export map in the package configuration to list a types condition first before import and require, using the package’s existing declaration output path.packages/voice-agents/tsconfig.cjs.json (1)
5-5: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueReplace deprecated
moduleResolution: "node"in the CJS tsconfig.TypeScript 6 marks
moduleResolution: "node"/"node10"as deprecated in favor ofmoduleResolution: "node16"/"node18"/NodeNext. This config usescommonjsbut currently setsmoduleResolution: "node", so change both CJK entries tomoduleResolution: "node16"and removeignoreDeprecations: "6.0".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/voice-agents/tsconfig.cjs.json` at line 5, Update packages/voice-agents/tsconfig.cjs.json to replace both moduleResolution: "node" entries with moduleResolution: "node16", and remove the ignoreDeprecations: "6.0" setting while preserving the existing CommonJS module configuration.packages/voice-agents/src/spoken-approval.ts (1)
157-170: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueThe ReDoS hint here is a false positive, but you can precompile the patterns.
phrasescomes from the module-levelGRANTSandDENIALSconstants. No user input reachesnew RegExp. The static analysis warning does not apply.takePhrasesstill builds 37RegExpobjects on every call. Build them once at module scope.♻️ Proposed refactor
-const takePhrases = (text: string, phrases: string[]) => { +const patternsFor = (phrases: string[]) => + phrases.map((phrase) => new RegExp(`\\b${phrase}\\b`, 'g')) + +const GRANT_PATTERNS = patternsFor(GRANTS) +const DENIAL_PATTERNS = patternsFor(DENIALS) + +const takePhrases = (text: string, patterns: RegExp[]) => { let found = false let remaining = text - for (const phrase of phrases) { + for (const pattern of patterns) { // Matched phrases are removed so a later pass cannot read the same words // again — this is what keeps "no problem" from also counting as a refusal. - const next = remaining.replace(new RegExp(`\\b${phrase}\\b`, 'g'), ' ') + pattern.lastIndex = 0 + const next = remaining.replace(pattern, ' ')🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/voice-agents/src/spoken-approval.ts` around lines 157 - 170, Precompile the phrase-matching regular expressions for the module-level GRANTS and DENIALS values instead of constructing them inside takePhrases on every call. Update takePhrases to reuse the prebuilt patterns while preserving phrase removal order and the existing found/remaining behavior.Source: Linters/SAST tools
packages/voice-agents/src/audio-playback-queue.ts (2)
86-93: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDocument that the returned promise rejects on decode failure.
drain()handles a failed decode internally, butenqueuereturns a derived promise frombuffer.then(...). A caller that writesvoid queue.enqueue(...)gets an unhandled rejection. The test file already works around this ataudio-playback-queue.test.tsLine 97. State the contract in the JSDoc, so callers know they must handle rejection.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/voice-agents/src/audio-playback-queue.ts` around lines 86 - 93, Update the JSDoc for enqueue in the audio playback queue to explicitly document that its returned promise resolves when decoding completes but rejects if audio decoding fails. Keep the existing enqueue and drain behavior unchanged, and make clear that callers must handle the rejection.
114-132: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winAn interrupt during playback leaves the
play()promise pending forever.
interrupt()setssource.onended = nulland callssource.stop(). ThePromisecreated inplay()then has no path toresolve. Thedrain()loop that awaits it stays suspended permanently and retains its closure. Each interruption during playback adds one such suspended loop.Keep a resolver for the current utterance and settle it in
interrupt().♻️ Proposed refactor
private current: { text: string source: AudioBufferSourceNode startedAt: number duration: number + done: () => void } | null = null- const { text, source, startedAt, duration } = this.current + const { text, source, startedAt, duration, done } = this.current complete = false source.onended = null try { source.stop() } catch { // Already stopped by the scheduler; nothing to unwind. } const elapsed = Math.max(0, this.ctx.currentTime - startedAt) const fraction = duration > 0 ? Math.min(1, elapsed / duration) : 0 const cut = text.slice(0, Math.round(text.length * fraction)).trimEnd() if (cut) heard.push(cut) this.current = null + done()this.current = { text: pending.text, source, startedAt: this.ctx.currentTime, duration: pending.buffer.duration, + done: resolve, }Also applies to: 180-200
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/voice-agents/src/audio-playback-queue.ts` around lines 114 - 132, Update the current utterance state in play() to retain its promise resolver, then invoke that resolver from interrupt() before clearing this.current so the interrupted playback promise settles and drain() can continue. Preserve normal onended resolution and ensure the resolver is settled only once.packages/voice-agents/src/use-voice-conversation.ts (1)
81-82: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMove the
optionsRef.currentwrite into an effect.This hook declares React 18/19 support, and mutating a ref during render can leave the ref pointing to an abandoned render in concurrent mode. Set
optionsRef.current = optionsinuseEffect, and add the necessary cleanup sostart/onSpeechStartdo not use stale options after unmount or dependency change.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/voice-agents/src/use-voice-conversation.ts` around lines 81 - 82, Move the optionsRef.current assignment out of render and into a useEffect within the voice conversation hook. Add effect cleanup that invalidates or clears the ref on unmount and before dependency changes, ensuring start and onSpeechStart cannot read stale options.packages/services/ai-deepinfra/run-tests.sh (1)
31-35: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse a null-delimited read to collect test files.
Word splitting on
findoutput breaks on paths that contain spaces or globbing characters.mapfilewith-print0removes that risk. Note also thatshopt -s nullglobon Line 7 and thepatternvariable are now only decorative, becausefindperforms the discovery.Static analysis reports the same concern (SC2207).
♻️ Proposed change
-# Expand the pattern into an array of files -files=($(find src -type f -name "*.test.ts")) +# Expand the pattern into an array of files +files=() +while IFS= read -r -d '' file; do + files+=("$file") +done < <(find src -type f -name "*.test.ts" -print0)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/services/ai-deepinfra/run-tests.sh` around lines 31 - 35, Update the test-file collection in the script to read null-delimited results from find using mapfile and find -print0, preserving paths containing spaces or glob characters. Remove the now-unused pattern variable and nullglob setting because discovery no longer relies on shell glob expansion.Source: Linters/SAST tools
packages/services/ai-deepinfra/src/transcription-model.ts (2)
179-183: 🗄️ Data Integrity & Integration | 🔵 Trivial | 💤 Low value
String(value)flattens object and array provider extras.A provider extra such as
{ deepinfra: { diarization: { speakers: 2 } } }becomes the multipart value[object Object]. The speech model sends extras as JSON, so the two models accept different extra shapes for the same option name. Serialise non-primitive values as JSON.♻️ Proposed change
for (const [key, value] of Object.entries( providerExtras(options.providerOptions) )) { - if (value !== undefined && value !== null) body.append(key, String(value)) + if (value === undefined || value === null) continue + body.append( + key, + typeof value === 'object' ? JSON.stringify(value) : String(value) + ) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/services/ai-deepinfra/src/transcription-model.ts` around lines 179 - 183, Update the provider extras loop in the transcription model to serialize object and array values with JSON while preserving direct string conversion for primitive values. Ensure multipart body fields receive valid JSON for non-primitive extras, including nested objects such as diarization configuration.
203-204: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winGive a non-JSON success response the same model-named error as a failure.
response.json()throws a bareSyntaxErrorwhen a 200 response carries HTML or plain text, for example from a proxy or an edge cache. The failure path on Line 201 names the model, andspeech-model.tsguards oncontent-typebefore it parses. This path does neither, so the caller loses the model id and the body.♻️ Proposed change
- const payload = (await response.json()) as DeepInfraTranscriptionResponse + const raw = await response.text() + let payload: DeepInfraTranscriptionResponse + try { + payload = JSON.parse(raw) as DeepInfraTranscriptionResponse + } catch { + throw new Error( + `DeepInfra transcription model '${this.modelId}' returned a non-JSON body: ${raw.slice(0, 500)}` + ) + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/services/ai-deepinfra/src/transcription-model.ts` around lines 203 - 204, Update the transcription response handling around response.json() to validate the response content type before parsing, and handle non-JSON success bodies with the same model-named error format as the existing failure path. Preserve the parsed payload flow for valid JSON responses while including the response body and model identifier when rejecting invalid success responses.packages/services/ai-deepinfra/package.json (1)
20-25: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAlign the release script with Yarn and consider exporting types in the exports map.
Two small points:
- Line 20 uses
npm run build && npm testwhile the rest of the monorepo uses Yarn.- The
exportsmap has notypescondition. Consumers that resolve withnode16/nodenextorbundlerreadexportsand can missdist/index.d.ts.♻️ Proposed change
- "release": "npm run build && npm test", + "release": "yarn build && yarn test", "prepublishOnly": "yarn build" }, "exports": { - ".": "./dist/index.js" + ".": { + "types": "./dist/index.d.ts", + "default": "./dist/index.js" + } },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/services/ai-deepinfra/package.json` around lines 20 - 25, Update the package.json release script to use the repository’s Yarn commands instead of npm, while preserving the existing build-and-test sequence. Extend the exports entry for "." with a types condition pointing to the generated declaration file at dist/index.d.ts alongside the existing runtime export.packages/services/ai-deepinfra/src/deepinfra-config.ts (2)
39-47: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueA differently cased
authorizationkey insettings.headersproduces two header entries.The returned object is a plain record. If a caller passes
headers: { Authorization: '…' }, the record holds bothauthorizationandAuthorization.fetchthen normalises both names into one field and joins the values with a comma, which yields an invalid credential rather than an override.Normalise the key names before the merge, or document that
settings.headersmust use lowercase names.♻️ Proposed change
headers: () => { const apiKey = settings.apiKey ?? process.env.DEEPINFRA_API_KEY if (!apiKey) { throw new Error( 'DeepInfra API key is missing. Pass `apiKey` to createDeepInfra(), or set DEEPINFRA_API_KEY.' ) } - return { authorization: `Bearer ${apiKey}`, ...settings.headers } + const extra = Object.fromEntries( + Object.entries(settings.headers ?? {}).map(([key, value]) => [ + key.toLowerCase(), + value, + ]) + ) + return { authorization: `Bearer ${apiKey}`, ...extra } },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/services/ai-deepinfra/src/deepinfra-config.ts` around lines 39 - 47, Update the headers callback in the DeepInfra configuration so settings.headers keys are normalised case-insensitively before merging with the generated authorization header, ensuring any casing of authorization overrides or is overridden without producing duplicate entries. Preserve all other custom headers unchanged.
48-48: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winBind the default
fetchtoglobalThis.
settings.fetch ?? globalThis.fetchstores a detached reference. Browsers reject detachedfetchcalls and throwTypeError: Illegal invocation, so the browser-facing@pikku/voice-agentspath can fail. ResolveglobalThis.fetchlazily and bind it.♻️ Proposed change
- fetch: settings.fetch ?? globalThis.fetch, + fetch: settings.fetch ?? ((...args) => globalThis.fetch(...args)),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/services/ai-deepinfra/src/deepinfra-config.ts` at line 48, Update the default fetch handling in the deepinfra configuration so that when settings.fetch is absent, globalThis.fetch is resolved lazily and bound to globalThis before storage or invocation; preserve any explicitly supplied settings.fetch unchanged.packages/services/ai-deepinfra/src/transcription-model.test.ts (1)
5-22: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
stubFetchis duplicated in both test files.
speech-model.test.tsLines 5-24 defines a near-identical helper that adds aUint8Arraybranch. Extract one shared helper, for examplesrc/test-fetch.ts, so both models test against the same stub behaviour. Thetsconfig.jsonexcludelist only removes**/*.test.ts, so name any shared helper accordingly or add it to the exclude list to keep it out ofdist.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/services/ai-deepinfra/src/transcription-model.test.ts` around lines 5 - 22, Extract the duplicated stubFetch helper from transcription-model.test.ts and speech-model.test.ts into one shared test-fetch helper, preserving the Uint8Array, string, JSON, status, headers, and call-capture behavior used by both tests. Import and reuse that helper in both test files, and name the shared file appropriately or update tsconfig.json so it remains excluded from dist.packages/services/ai-deepinfra/src/speech-model.test.ts (1)
101-143: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider covering
abortSignaland per-call header merging for the speech model.
transcription-model.test.tsverifies thatabortSignalreachesfetchon Lines 227-239. The speech model forwards the same signal onspeech-model.tsLine 78, and it mergesdefinedHeaders(options.headers)on Line 60, but neither path has a test here. Barge-in depends on the signal, so a regression would be silent.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/services/ai-deepinfra/src/speech-model.test.ts` around lines 101 - 143, Add speech-model tests covering both per-call abortSignal propagation to fetch and per-call header merging with the configured headers. Use the existing stubFetch and speech model setup, assert the fetch receives the exact signal, and verify custom defined headers are included alongside the model’s default headers.packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts (1)
289-319: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRemove the
call.mutatingflag or make it change state.
call.mutatingis initialised totrueat Line 295, andonBeginChangesat Line 303-305 sets it totrueagain. ThecollectResultgetter at Line 312-314 therefore always returnstrue. The flag and the hook have no effect.The conservative default is correct and is documented in
beginChanges. The suppression for a tool that stops at its checkpoint already comes from theAbandonedErrorbranch at Line 179. Drop the flag so the code states what it does.♻️ Proposed simplification
execute: (input: unknown) => { // Assumed to have changed something unless it says otherwise — the tool // that never declares its checkpoint is the one that cannot be assumed // harmless. `beginChanges()` can only ever make this more precise. - const call = { mutating: true } const scope: AbortScope = { get abandoned() { return handle.signal.aborted }, get reason() { return handle.interruption?.reason }, - onBeginChanges: () => { - call.mutating = true - }, } return handle.trackTool( tool.name, randomUUID(), - () => runInAbortScope(scope, () => tool.execute(input)), - { - get collectResult() { - return call.mutating - }, - } + () => runInAbortScope(scope, () => tool.execute(input)) ) },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts` around lines 289 - 319, Remove the redundant call.mutating state and the onBeginChanges hook from the execute wrapper in the tool mapping. Keep collectResult as an unconditional true value, preserving the documented conservative default and existing AbandonedError suppression behavior.packages/core/src/wirings/ai-agent/voice-output.ts (1)
252-261: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDo not log an aborted synthesis as an error.
On barge-in the run aborts, and
synthesizeAudiorejects for every sentence still in flight. Each rejection reaches this branch and writes alogger.errorline. A normal interruption then produces error-level noise proportional to the number of queued sentences. Distinguish the abort from a provider failure.♻️ Proposed change
state.tail = (state.tail ?? Promise.resolve()).then(async () => { const result = await synthesis if (!result.ok) { + // An interrupt cancels every sentence still in flight. That is the + // feature working, not a provider failure. + if (signal?.aborted) return // One sentence going unspoken is better than the rest of the reply // never arriving, so the chain carries on. logger?.error( `voiceOutput could not synthesize a sentence: ${String(result.error)}` ) return }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/core/src/wirings/ai-agent/voice-output.ts` around lines 252 - 261, Update the error handling around the synthesis result in the state.tail chain to distinguish an expected run abort from a provider failure. Suppress logger.error for abort-related rejections during barge-in, while preserving the existing error log and chain continuation for genuine synthesis failures.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@e2e/packages/functions/src/agents/voice.agent.ts`:
- Around line 9-13: Resolve the model inconsistency in the voice agent’s
aiMiddleware configuration: use deepinfra/openai/whisper-large-v3-turbo as the
primary voiceInput model unless Nemotron is intentionally required here. If
retaining Nemotron, add a brief comment explaining why voice.agent differs from
voice-assistant.agent and voice-demo.function.
In `@e2e/packages/functions/src/functions/voice-demo.function.ts`:
- Around line 748-755: Guard inFlightTools before accessing its length in the
interrupt handling logic after post('/voice-demo/interrupt'). Preserve the
existing note behavior when stopped is true and tools are present, while
treating an omitted or undefined inFlightTools value as an empty list.
- Around line 880-885: Update the startSession error path around the shown catch
block to destroy the existing AudioPlaybackQueue before clearing playback.
Invoke playback.destroy() when the queue was created, then set playback to null
while preserving the existing session reset and idle-state behavior.
- Around line 304-318: The voiceDemoInterrupt route currently claims
rpc.agent.interrupt performs caller ownership validation despite being
sessionless with auth disabled. Update the route to establish and pass the demo
owner/principal before calling rpc.agent.interrupt, or restrict interruption to
runs verified as owned during stream runtime; otherwise remove the inaccurate
ownership assertion from the function comment.
- Around line 44-93: Update the asset resolution used with SERVABLE and distOf
so each package group resolves from the correct asset directory rather than
assuming the package entry-point directory. Use package-specific resolve
targets: the voice assets from the voice package, vad-web’s root-level bundle
assets from its package root, and onnxruntime-web’s listed runtime assets from
its dist directory; preserve the existing allow-list and clear failure behavior
for unresolved assets.
In `@e2e/packages/functions/tests/scenarios/converse.steps.ts`:
- Around line 131-155: Update the matches filter in the todo assertion to use
case-insensitive exact title equality instead of substring matching. Preserve
the existing no-match and duplicate diagnostics, ensuring titles such as “get
lunch ingredients” do not match “get lunch.”
In `@packages/core/src/function/function-runner.ts`:
- Around line 294-299: Update the invocation setup around
invocationWire.beginChanges to capture its previous value, assign it only for
non-readonly functions, and delete it for readonly functions. Restore the
original beginChanges value in both existing finally blocks, following the
handling of functionId, audit, and addonNamespace, so reused wires retain their
prior state.
In `@packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts`:
- Around line 331-346: Update trackInterruptNote so the tracked promise handles
write failures internally: catch rejections from write/finally, log the error
using the module’s established logging mechanism, and ensure the promise stored
in pendingNotes resolves after logging. Preserve the existing cleanup that
removes the entry only when it still matches the tracked promise, so
awaitPendingInterruptNote does not propagate storage failures into the next run.
In `@packages/core/src/wirings/ai-agent/ai-agent-prepare.ts`:
- Around line 1027-1031: Update the pending interrupt-note wait in
prepareAgentRun to catch failures from awaitPendingInterruptNote(threadId), log
them as note-persistence failures, and allow preparation to continue so
streamAIAgent/runAIAgent retain their normal error/done handling. Keep
successful waiting behavior unchanged.
In `@packages/core/src/wirings/ai-agent/ai-agent-runner.ts`:
- Around line 381-396: Preserve accumulated usage when interrupted in both run
paths. In packages/core/src/wirings/ai-agent/ai-agent-runner.ts at lines 381-396
and 861-874, hoist totalUsage above the try blocks in runAIAgent and
continueAfterToolResultSync, then include usage with totalUsage and agent.model
in each interrupted updateRun call; apply the same change at both sites.
In `@packages/core/src/wirings/ai-agent/ai-agent.types.ts`:
- Around line 63-79: Update all AI storage implementations used by
convertToSDKMessages so saveMessages persists the interrupted and undelivered
fields, and getMessages restores them on returned messages. Preserve these
optional flags unchanged across writes and reads, including false/undefined
behavior, so assistant and tool messages retain their interrupt metadata.
In `@packages/core/src/wirings/ai-agent/voice-input.ts`:
- Around line 130-139: Update the voice listening loop around
streamAIAgent/runAIAgent to catch NoSpeechDetectedError from
voiceInput.modifyInput, resume listening without forwarding the error to the
agent transport or reporting a failed run, and preserve existing handling for
all other errors.
In `@packages/core/src/wirings/ai-agent/voice-output.ts`:
- Around line 270-282: Bound the await on state.tail in the event.type ===
'done' branch so a stalled synthesis call cannot prevent completion
indefinitely. Race the pending tail against a finite deadline and continue
clearing state.tail and returning the audio-done event when the deadline
expires, while preserving the existing behavior when the tail settles normally.
- Around line 43-54: Update the SCRIPT_RANGES.latin pattern to match only Latin
script characters, excluding punctuation and symbols currently covered by the
broad A–ɏ range. Prefer a Unicode Script-based expression if supported, while
preserving the behavior of the other script patterns.
In `@packages/services/ai-vercel/src/vercel-ai-agent-runner.ts`:
- Around line 9-16: Add `@ai-sdk/provider` as a direct dependency or
peerDependency in packages/services/ai-vercel/package.json, matching the
existing import of SharedV3ProviderOptions in vercel-ai-agent-runner.ts and
using a compatible version with the installed ai package.
In `@packages/voice-agents/src/audio-playback-queue.ts`:
- Around line 140-153: Update destroy() and the interrupt/resume flow so the
asynchronous resume triggered by interrupt() cannot become an unhandled
rejection when ctx.close() runs; either await and handle the resume before
closing the AudioContext or attach an explicit rejection handler, while
preserving the queue’s existing interruption behavior.
In `@packages/voice-agents/src/speech-detector.ts`:
- Around line 262-277: Update the stop method returned by the speech detector
setup to attach a rejection handler to vad.destroy(). Route any destruction
error through handlers.onError, preserving the existing stopped guard and
fire-and-forget behavior.
In `@packages/voice-agents/src/spoken-approval.ts`:
- Around line 131-147: Update interpretConsent and its phrase matching around
DENIALS/takePhrases so negated grant phrases such as “not okay” and “that is not
fine” return “denied” rather than “granted”; detect “not” before a grant phrase
or include it in the denial handling, and add the specified covering assertions
to spoken-approval.test.ts.
In `@packages/voice-agents/src/voice-session.ts`:
- Around line 21-24: Update the silenceDelay documentation in the voice session
options to state the actual default of 700 milliseconds, matching the fallback
used by captureNextTurn; leave the minDecibels documentation unchanged.
- Around line 134-150: Ensure the start() flow arms the speech detector at most
once per acquired stream by guarding startSpeechDetector() with the existing
detector state, such as this.stopSpeech. Preserve detector reuse across stop()
followed by start(), while still starting it when no detector has been created.
---
Nitpick comments:
In `@e2e/packages/functions/src/agents/voice-assistant.agent.ts`:
- Around line 120-150: Centralize the voice configuration currently passed to
voiceInput and voiceOutput in this agent, including the ASR model id, TTS model
id, and speakableScripts map, as exported constants. Update both the voice
assistant agent and voice-demo.function.ts to import and use those shared
constants, removing the duplicate declarations while preserving the existing
model and voice values.
In `@e2e/packages/functions/src/functions/voice-demo.function.ts`:
- Around line 463-468: Update toBase64 to avoid byte-by-byte string
concatenation: convert the Uint8Array to binary in chunks before calling btoa,
preserving the same Base64 output and async behavior used by handleTurn.
- Around line 144-148: Bound the unauthenticated provider inputs by adding
generous maximum-length validation to the audio field in
VoiceDemoTranscribeInput and the text field in the corresponding text-input
schema. Apply the limits to the z.string() definitions before payloads reach
Buffer.from or the billed provider, preserving the existing schema behavior for
valid short inputs.
In `@packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts`:
- Around line 289-319: Remove the redundant call.mutating state and the
onBeginChanges hook from the execute wrapper in the tool mapping. Keep
collectResult as an unconditional true value, preserving the documented
conservative default and existing AbandonedError suppression behavior.
In `@packages/core/src/wirings/ai-agent/voice-output.ts`:
- Around line 252-261: Update the error handling around the synthesis result in
the state.tail chain to distinguish an expected run abort from a provider
failure. Suppress logger.error for abort-related rejections during barge-in,
while preserving the existing error log and chain continuation for genuine
synthesis failures.
In `@packages/core/src/wirings/http/web-request.test.ts`:
- Around line 354-369: Extend the binary-body test around applyWebResponse to
use a createMockResponse variant that defines send, then assert send receives
the original byte sequence unchanged. Preserve the existing arrayBuffer
assertion and verify the adapter-provided send branch with the same binary
payload.
In `@packages/services/ai-deepinfra/package.json`:
- Around line 20-25: Update the package.json release script to use the
repository’s Yarn commands instead of npm, while preserving the existing
build-and-test sequence. Extend the exports entry for "." with a types condition
pointing to the generated declaration file at dist/index.d.ts alongside the
existing runtime export.
In `@packages/services/ai-deepinfra/run-tests.sh`:
- Around line 31-35: Update the test-file collection in the script to read
null-delimited results from find using mapfile and find -print0, preserving
paths containing spaces or glob characters. Remove the now-unused pattern
variable and nullglob setting because discovery no longer relies on shell glob
expansion.
In `@packages/services/ai-deepinfra/src/deepinfra-config.ts`:
- Around line 39-47: Update the headers callback in the DeepInfra configuration
so settings.headers keys are normalised case-insensitively before merging with
the generated authorization header, ensuring any casing of authorization
overrides or is overridden without producing duplicate entries. Preserve all
other custom headers unchanged.
- Line 48: Update the default fetch handling in the deepinfra configuration so
that when settings.fetch is absent, globalThis.fetch is resolved lazily and
bound to globalThis before storage or invocation; preserve any explicitly
supplied settings.fetch unchanged.
In `@packages/services/ai-deepinfra/src/speech-model.test.ts`:
- Around line 101-143: Add speech-model tests covering both per-call abortSignal
propagation to fetch and per-call header merging with the configured headers.
Use the existing stubFetch and speech model setup, assert the fetch receives the
exact signal, and verify custom defined headers are included alongside the
model’s default headers.
In `@packages/services/ai-deepinfra/src/transcription-model.test.ts`:
- Around line 5-22: Extract the duplicated stubFetch helper from
transcription-model.test.ts and speech-model.test.ts into one shared test-fetch
helper, preserving the Uint8Array, string, JSON, status, headers, and
call-capture behavior used by both tests. Import and reuse that helper in both
test files, and name the shared file appropriately or update tsconfig.json so it
remains excluded from dist.
In `@packages/services/ai-deepinfra/src/transcription-model.ts`:
- Around line 179-183: Update the provider extras loop in the transcription
model to serialize object and array values with JSON while preserving direct
string conversion for primitive values. Ensure multipart body fields receive
valid JSON for non-primitive extras, including nested objects such as
diarization configuration.
- Around line 203-204: Update the transcription response handling around
response.json() to validate the response content type before parsing, and handle
non-JSON success bodies with the same model-named error format as the existing
failure path. Preserve the parsed payload flow for valid JSON responses while
including the response body and model identifier when rejecting invalid success
responses.
In `@packages/voice-agents/package.json`:
- Around line 10-15: Add the package-level types field pointing to the emitted
declarations, and update the "." export map in the package configuration to list
a types condition first before import and require, using the package’s existing
declaration output path.
In `@packages/voice-agents/README.md`:
- Around line 51-58: Update the “What each piece is for” export table to include
spokenApproval, interpretConsent, and detectSpeech, with concise purposes
matching their documented behavior and the existing README terminology.
In `@packages/voice-agents/run-tests.sh`:
- Line 4: Harden the run-tests.sh setup and file collection: update the cd
command near script_dir to fail immediately if changing directories fails, and
replace the unquoted find-based array assignment near files with whitespace-safe
path handling so filenames containing spaces remain single elements.
In `@packages/voice-agents/src/audio-playback-queue.ts`:
- Around line 86-93: Update the JSDoc for enqueue in the audio playback queue to
explicitly document that its returned promise resolves when decoding completes
but rejects if audio decoding fails. Keep the existing enqueue and drain
behavior unchanged, and make clear that callers must handle the rejection.
- Around line 114-132: Update the current utterance state in play() to retain
its promise resolver, then invoke that resolver from interrupt() before clearing
this.current so the interrupted playback promise settles and drain() can
continue. Preserve normal onended resolution and ensure the resolver is settled
only once.
In `@packages/voice-agents/src/speech-detector.ts`:
- Around line 156-277: Add speech-detector.test.ts covering detectSpeech
callbacks for speech starts, speech ends, misfires, redemptionMs-adjusted
timing, and idempotent stop() using mocked VadModule/VadInstance; add
silence-detector.test.ts covering detectSilence audible-to-silent transitions,
PROGRESS_STEP progress throttling, and idempotent stop() using mocked
AudioContext, AudioNode, and requestAnimationFrame. Update only the tests; no
direct production-code change is required in either detector.
In `@packages/voice-agents/src/spoken-approval.ts`:
- Around line 157-170: Precompile the phrase-matching regular expressions for
the module-level GRANTS and DENIALS values instead of constructing them inside
takePhrases on every call. Update takePhrases to reuse the prebuilt patterns
while preserving phrase removal order and the existing found/remaining behavior.
In `@packages/voice-agents/src/use-voice-conversation.ts`:
- Around line 81-82: Move the optionsRef.current assignment out of render and
into a useEffect within the voice conversation hook. Add effect cleanup that
invalidates or clears the ref on unmount and before dependency changes, ensuring
start and onSpeechStart cannot read stale options.
In `@packages/voice-agents/tsconfig.cjs.json`:
- Line 5: Update packages/voice-agents/tsconfig.cjs.json to replace both
moduleResolution: "node" entries with moduleResolution: "node16", and remove the
ignoreDeprecations: "6.0" setting while preserving the existing CommonJS module
configuration.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 07cbdd15-a1a3-478e-903a-f2d8953a990e
⛔ Files ignored due to path filters (1)
yarn.lockis excluded by!**/yarn.lock,!**/*.lock
📒 Files selected for processing (75)
e2e/package.jsone2e/packages/functions/package.jsone2e/packages/functions/src/agents/voice-assistant.agent.tse2e/packages/functions/src/agents/voice.agent.tse2e/packages/functions/src/functions/voice-demo.function.tse2e/packages/functions/src/wirings/voice-demo.wirings.tse2e/packages/functions/tests/scenarios/agent-assert.steps.tse2e/packages/functions/tests/scenarios/converse.steps.tse2e/packages/functions/tests/scenarios/voice-approval.feature.tse2e/packages/functions/tests/scenarios/voice-assistant-converse.feature.tse2e/packages/todos/src/functions/getTodo.function.tse2e/packages/todos/src/functions/listTodos.function.tse2e/pikku.config.jsone2e/src/services.tspackages/cli/src/functions/commands/dev-ai-runner.tspackages/cli/src/functions/commands/scenario.tspackages/cli/types/config.d.tspackages/core/src/function/abort-scope.test.tspackages/core/src/function/abort-scope.tspackages/core/src/function/function-runner.tspackages/core/src/index.tspackages/core/src/services/ai-agent-runner-service.tspackages/core/src/types/core.types.tspackages/core/src/wirings/actor-flow/run-conversation.tspackages/core/src/wirings/ai-agent/ai-agent-agui.test.tspackages/core/src/wirings/ai-agent/ai-agent-agui.tspackages/core/src/wirings/ai-agent/ai-agent-helpers.tspackages/core/src/wirings/ai-agent/ai-agent-interrupt.test.tspackages/core/src/wirings/ai-agent/ai-agent-interrupt.tspackages/core/src/wirings/ai-agent/ai-agent-prepare.tspackages/core/src/wirings/ai-agent/ai-agent-runner.tspackages/core/src/wirings/ai-agent/ai-agent-stream.test.tspackages/core/src/wirings/ai-agent/ai-agent-stream.tspackages/core/src/wirings/ai-agent/ai-agent.types.tspackages/core/src/wirings/ai-agent/index.tspackages/core/src/wirings/ai-agent/voice-input.test.tspackages/core/src/wirings/ai-agent/voice-input.tspackages/core/src/wirings/ai-agent/voice-output.test.tspackages/core/src/wirings/ai-agent/voice-output.tspackages/core/src/wirings/http/http.types.tspackages/core/src/wirings/http/web-request.test.tspackages/core/src/wirings/http/web-request.tspackages/core/src/wirings/rpc/rpc-runner.tspackages/core/src/wirings/rpc/rpc-types.tspackages/services/ai-deepinfra/LICENSEpackages/services/ai-deepinfra/README.mdpackages/services/ai-deepinfra/package.jsonpackages/services/ai-deepinfra/run-tests.shpackages/services/ai-deepinfra/src/deepinfra-config.tspackages/services/ai-deepinfra/src/index.tspackages/services/ai-deepinfra/src/speech-model.test.tspackages/services/ai-deepinfra/src/speech-model.tspackages/services/ai-deepinfra/src/transcription-model.test.tspackages/services/ai-deepinfra/src/transcription-model.tspackages/services/ai-deepinfra/tsconfig.jsonpackages/services/ai-vercel/src/message-converter.test.tspackages/services/ai-vercel/src/message-converter.tspackages/services/ai-vercel/src/vercel-ai-agent-runner.test.tspackages/services/ai-vercel/src/vercel-ai-agent-runner.tspackages/services/kysely/src/kysely-tables.tspackages/voice-agents/README.mdpackages/voice-agents/package.jsonpackages/voice-agents/run-tests.shpackages/voice-agents/src/audio-playback-queue.test.tspackages/voice-agents/src/audio-playback-queue.tspackages/voice-agents/src/index.tspackages/voice-agents/src/silence-detector.tspackages/voice-agents/src/speech-detector.tspackages/voice-agents/src/spoken-approval.test.tspackages/voice-agents/src/spoken-approval.tspackages/voice-agents/src/use-voice-conversation.tspackages/voice-agents/src/voice-session.tspackages/voice-agents/tsconfig.cjs.jsonpackages/voice-agents/tsconfig.jsonverifiers/middleware-and-permissions/src/start.ts
| aiMiddleware: [ | ||
| voiceInput({ | ||
| model: 'deepinfra/nvidia/Nemotron-3.5-ASR-Streaming-Multilingual-0.6b', | ||
| }), | ||
| ], |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Confirm that Nemotron is intended here.
voice-assistant.agent.ts records that Nemotron truncates the tail of short clips and that Whisper replaced it for that reason. voice-demo.function.ts also keeps Nemotron only as a comparison model. This agent now adopts Nemotron as its primary transcription model, which points the opposite way. If the choice is deliberate, add a short comment stating why this agent differs. If not, use deepinfra/openai/whisper-large-v3-turbo here too.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@e2e/packages/functions/src/agents/voice.agent.ts` around lines 9 - 13,
Resolve the model inconsistency in the voice agent’s aiMiddleware configuration:
use deepinfra/openai/whisper-large-v3-turbo as the primary voiceInput model
unless Nemotron is intentionally required here. If retaining Nemotron, add a
brief comment explaining why voice.agent differs from voice-assistant.agent and
voice-demo.function.
| const distOf = (specifier: string) => | ||
| dirname(fileURLToPath(import.meta.resolve(specifier))) | ||
|
|
||
| /** | ||
| * Files the page may load, and which package each comes from. | ||
| * | ||
| * An allow-list rather than path sanitising: this route reads from disk under a | ||
| * name that arrives in a URL, and the set of files a demo needs is small, | ||
| * known, and never user-supplied. | ||
| * | ||
| * The first group is the voice package itself, imported as modules — | ||
| * `voice-session.js` pulls in the two detectors by relative import, so they | ||
| * have to be here too. The rest exist only for the Silero turn detector, which | ||
| * is opt-in on the page and served locally rather than from the library's CDN | ||
| * so it can be tried offline and so the ~2MB model download is visible in the | ||
| * network tab where it belongs. | ||
| * | ||
| * `@ricky0123/vad-web`'s browser bundle is UMD expecting a global `ort`, which | ||
| * is why onnxruntime's classic-script build is here rather than its ESM one. | ||
| * Its `.mjs`/`.wasm` pairs are fetched by onnxruntime itself at runtime, not by | ||
| * the page — which variant depends on what the browser supports, so all of | ||
| * them are listed. | ||
| */ | ||
| const SERVABLE = new Map<string, string>([ | ||
| ...( | ||
| [ | ||
| 'voice-session.js', | ||
| 'silence-detector.js', | ||
| 'speech-detector.js', | ||
| 'audio-playback-queue.js', | ||
| 'spoken-approval.js', | ||
| ] as const | ||
| ).map((file) => [file, '@pikku/voice-agents'] as [string, string]), | ||
| ...( | ||
| [ | ||
| 'bundle.min.js', | ||
| 'vad.worklet.bundle.min.js', | ||
| 'silero_vad_v5.onnx', | ||
| ] as const | ||
| ).map((file) => [file, '@ricky0123/vad-web'] as [string, string]), | ||
| ...( | ||
| [ | ||
| 'ort.wasm.min.js', | ||
| 'ort-wasm-simd-threaded.mjs', | ||
| 'ort-wasm-simd-threaded.wasm', | ||
| 'ort-wasm-simd-threaded.jsep.mjs', | ||
| 'ort-wasm-simd-threaded.jsep.wasm', | ||
| ] as const | ||
| ).map((file) => [file, 'onnxruntime-web'] as [string, string]), | ||
| ]) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Locate the resolved entry points and the requested asset files for each package.
set -euo pipefail
for pkg in "`@ricky0123/vad-web`" "onnxruntime-web" "`@pikku/voice-agents`"; do
echo "=== $pkg ==="
fd -H -t f 'package.json' node_modules/"$pkg" --max-depth 1 --exec jq '{name, main, module, exports}' {} \; 2>/dev/null || echo "not installed in sandbox"
done
echo "=== asset locations ==="
for f in bundle.min.js vad.worklet.bundle.min.js silero_vad_v5.onnx ort.wasm.min.js ort-wasm-simd-threaded.mjs; do
fd -H -t f "^${f}$" node_modules 2>/dev/null | head -5
doneRepository: pikkujs/pikku
Length of output: 329
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== locate voice-demo function ==="
fd -H -t f 'voice-demo\.function\.ts$' . || true
file="$(fd -H -t f 'voice-demo\.function\.ts$' . | head -1)"
if [ -n "${file:-}" ]; then
echo "=== outline ==="
ast-grep outline "$file" --view expanded || true
echo "=== lines 1-140 ==="
sed -n '1,140p' "$file" | nl -ba
fi
echo "=== package lock excerpts ==="
if [ -f yarn.lock ]; then
rg -n '"(`@ricky0123/vad-web`"|"onnxruntime-web"|"`@pikku/voice-agents`"|`@ricky0123/vad-web`|onnxruntime-web|`@pikku/voice-agents`)' yarn.lock | head -80
fi
for f in pnpm-lock.yaml package-lock.json yarn.lock; do
[ -f "$f" ] && echo "--- $f package refs ---" && rg -n -F '`@ricky0123/vad-web`' -F 'onnxruntime-web' -F '`@pikku/voice-agents`' "$f" | head -80 || true
doneRepository: pikkujs/pikku
Length of output: 739
🌐 Web query:
@ricky0123/vad-web package.json main exports bundle.min.js vad.worklet.bundle.min.js silero_vad_v5.onnx onnxruntime-web package.json exports ort.wasm.min.js
💡 Result:
The @ricky0123/vad-web package is a client-side voice activity detection (VAD) library for the browser that relies on ONNX Runtime Web for model execution [1][2]. Key file and asset details for integration include: 1. Bundle Files: The primary library entry point for browser script tags is dist/bundle.min.js [3][4]. For its internal processing, the library requires a Web Audio API worklet, located at dist/vad.worklet.bundle.min.js [2][5]. 2. Model Files: The package includes Silero VAD models in ONNX format, specifically silero_vad_v5.onnx and a legacy version, silero_vad_legacy.onnx [2][5]. These must be accessible to the library at runtime [2]. 3. ONNX Runtime Web: The library requires onnxruntime-web as a dependency [1][2]. When using script tags, you must load the ONNX Runtime Web entry point, typically ort.js or similar, and ensure the WebAssembly binary (ort.wasm.min.js) is available for the library to load [1][4][2]. 4. Configuration: When initializing the library, you must provide base paths so it can locate these assets [2][5]: - baseAssetPath: The directory where dist/vad.worklet.bundle.min.js, silero_vad_v5.onnx, and related assets are hosted [2][5]. - onnxWASMBasePath: The directory where the onnxruntime-web WASM files (including ort.wasm.min.js) are hosted [2][5]. If you are using a bundler (like Vite or Webpack), you will typically need to copy these specific files from the node_modules/@ricky0123/vad-web/dist/ and node_modules/onnxruntime-web/dist/ directories into your public or output folder so they are accessible at the specified paths at runtime [2].
Citations:
- 1: https://www.npmjs.com/package/@ricky0123/vad-web
- 2: https://docs.vad.ricky0123.com/user-guide/browser/
- 3: https://registry.npmjs.org/%40ricky0123%2Fvad-web
- 4: https://github.com/ricky0123/vad/
- 5: https://docs.vad.ricky0123.com/user-guide/api/
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="$(fd -H -t f 'voice-demo\.function\.ts$' . | head -1)"
echo "=== file ==="
printf '%s\n' "$file"
echo "=== lines 1-140 ==="
sed -n '1,140p' "$file" | awk '{printf "%4d\t%s\n", NR, $0}'
echo "=== package manager files ==="
git ls-files | egrep '(^|/)(pnpm-lock\.yaml|package-lock\.json|yarn\.lock|package\.json)$' | head -50Repository: pikkujs/pikku
Length of output: 8571
🌐 Web query:
registry.npmjs.org @ricky0123/vad-web 1.5.4 package.json exports
💡 Result:
As of August 2, 2026, the package @ricky0123/vad-web does not have a version 1.5.4; the latest available version is 0.0.30 [1][2][3]. Furthermore, the @ricky0123/vad-web package does not define an exports field in its package.json [4][5][6]. Instead of using the exports field, the package exposes its functionality primarily through the main entry point and direct imports from the dist/ directory [4][5][6]. Common patterns for importing from this package include: import { MicVAD } from "@ricky0123/vad-web"; // Or, for specific components: import { MicVAD } from "@ricky0123/vad-web/dist/index"; [7] The library's structure relies on a flat set of files within the dist/ folder, including index.js, real-time-vad.js, and various worker/model bundles [4]. Developers are typically advised to reference these files directly or import from the root package path, and static assets (like.onnx model files and.wasm files) must often be manually managed and served by the application's build system (e.g., Vite or Webpack) [8][9][7].
Citations:
- 1: https://www.npmjs.com/package/@ricky0123/vad-web
- 2: https://npmx.dev/package/@ricky0123/vad-web/versions
- 3: https://packages.ecosyste.ms/registries/npmjs.org/packages/@ricky0123%2Fvad-web
- 4: https://registry.npmjs.org/%40ricky0123%2Fvad-web
- 5: https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.18/dist/index.d.ts
- 6: https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.30/dist/real-time-vad.d.ts
- 7: Trying to get this bundled into a vite/SvelteKit app ricky0123/vad#34
- 8: https://docs.vad.ricky0123.com/user-guide/browser/
- 9: https://github.com/ricky0123/vad
🌐 Web query:
registry.npmjs.org onnxruntime-web package.json exports default import main "ort.wasm.min.js" "ort-wasm-simd-threaded.mjs"
💡 Result:
In the onnxruntime-web package, the package.json file uses the exports field to define conditional entry points for various environments and bundling strategies [1]. When importing onnxruntime-web in a JavaScript environment, the specific file loaded depends on your import style (ESM vs. CommonJS) and the target platform [1]. The files you mentioned, ort.wasm.min.js, ort-wasm-simd-threaded.mjs, and ort-wasm-simd-threaded.wasm, are core components of the WebAssembly execution provider [2][3]. Key details: 1. Entry Points and Imports: - When using standard imports (e.g., import * as ort from 'onnxruntime-web'), the package resolves to dist/ort.bundle.min.mjs (for ESM) or dist/ort.min.js (for CommonJS) by default [1]. - You can specifically target the WebAssembly-only runtime by importing from onnxruntime-web/wasm [1]. 2. Role of WASM Files: - ort-wasm-simd-threaded.mjs and ort-wasm-simd-threaded.wasm are the default WebAssembly backend artifacts [3]. As of version 1.19.0, multi-threaded SIMD is the standard and only supported WASM build, replacing previous non-threaded variants [3]. - ort.wasm.min.js acts as the JavaScript entry point for the WASM backend, often used in scenarios like Web Workers where scripts need to be loaded via importScripts() or direct script tags [4][5]. 3. Usage and Configuration: - These files are typically located in the dist/ directory of the package [1][4]. - If you are hosting the files yourself (rather than using a CDN), you must ensure that the onnxruntime-web library can locate these artifacts at runtime [6][5]. You can configure this using ort.env.wasm.wasmPaths to point to the directory where you have deployed the .wasm and .mjs files [5][7]. - For applications using bundlers like Vite or Webpack, the bundler often handles these assets automatically; however, if you are working with Web Workers or custom setups, you may need to manually copy these files to your public static assets folder to ensure they are accessible at the correct URL paths [4][5]. In summary, ort-wasm-simd-threaded.mjs and its accompanying .wasm file provide the high-performance inference engine, while ort.wasm.min.js serves as the loader/entry point for the WASM provider [4][3]. Direct interaction with these files is generally handled by the library itself, provided they are correctly discoverable in your environment [5][7].
Citations:
- 1: https://github.com/microsoft/onnxruntime/blob/master/js/web/package.json
- 2: https://onnxruntime.ai/docs/build/web.html
- 3: [Web] Clarification on wasm/simd vs wasm/simd/threaded default in onnxruntime-web v1.19.0+ microsoft/onnxruntime#25666
- 4: https://github.com/kizuna-ai-lab/sokuji/blob/05c8f89f/scripts/copy-ort-wasm.sh
- 5: https://github.com/stolinski/booth
- 6: https://github.com/keyur2maru/vad
- 7: https://onnxruntime.ai/docs/tutorials/web/env-flags-and-session-options.html
🌐 Web query:
registry.npmjs.org onnxruntime-web 1.19.0 package.json exports import default types
💡 Result:
In onnxruntime-web version 1.19.0, the package.json file utilizes the "exports" field to manage conditional imports, allowing developers to import specific subsets of the library based on their execution provider or environment needs [1][2]. The types for the package are provided via the "types" property within each entry of the "exports" field, pointing to the root "types.d.ts" file [1]. When importing the package, you can use either ES Modules (recommended) or CommonJS syntax [3][4]. The "exports" field defines the following mappings [1][2]: 1. Default Import (onnxruntime-web): Accessible via import * as ort from 'onnxruntime-web' (ESM) or const ort = require('onnxruntime-web') (CJS) [3][2][4]. 2. Conditional Sub-paths: You can reduce bundle sizes by importing only necessary features, such as: - WebGPU: onnxruntime-web/webgpu [3][4] - WebGL: onnxruntime-web/webgl [4] - Wasm: onnxruntime-web/wasm [4] - Experimental: onnxruntime-web/experimental [3][4] - Training: onnxruntime-web/training [4] The package structure within "exports" explicitly separates "import" (for ESM) and "require" (for CJS) conditions, and often provides bundles for different environments (e.g., node, web) [1]. Since version 1.19, the library optimized its module export to better support these modular imports [2]. If you are using TypeScript, the types are automatically resolved through the "types" definition in the "exports" mapping [1].
Citations:
- 1: https://github.com/microsoft/onnxruntime/blob/master/js/web/package.json
- 2: microsoft/onnxruntime@036fcd9
- 3: https://onnxruntime.ai/docs/get-started/with-javascript/web.html
- 4: https://github.com/microsoft/onnxruntime-inference-examples/tree/main/js/importing_onnxruntime-web
🌐 Web query:
registry.npmjs.org onnxruntime-web 1.19.0 package.json
💡 Result:
The onnxruntime-web version 1.19.0 package.json file, hosted on registry.npmjs.org, defines the library's metadata and structure for the ONNX Runtime Web v1.19.0 release (published August 2024) [1]. The package.json specifies the entry points and exports for the library, facilitating its use across Node.js and browser environments [2]. Key features of the package.json structure for this version include: Main and Browser Entry Points: - main: "dist/ort.node.min.js" - browser: "dist/ort.min.js" Exports Mapping: The "exports" field provides conditional exports to handle different module systems (ESM and CJS) and environments: - Root ("."): Configured with specific paths for node (import/require), and ESM import configurations including "onnxruntime-web-use-extern-wasm" and a default bundle ("dist/ort.bundle.min.mjs") [2]. - Additional entry points: "./all", "./wasm", "./webgl", and "./webgpu" are also defined to allow selective importing of components, such as specific execution providers [2]. Release Context: Version 1.19.0 introduced updates to JavaScript packaging to align with modern best practices, which notably included slight incompatibilities for applications bundling the library [1]. This release also necessitated specific runtime files (e.g.,.mjs files in the /dist directory) that caused integration issues for some CDNs like cdnjs that had not yet indexed the full file set [3]. You can view the raw content of the package.json file for this specific version directly via UNPKG at: https://app.unpkg.com/onnxruntime-web@1.19.0/files/package.json [4].
Citations:
- 1: https://github.com/Microsoft/onnxruntime/releases/tag/v1.19.0
- 2: https://github.com/microsoft/onnxruntime/blob/master/js/web/package.json
- 3: [Web] requested dist/*.mjs files for cdnjs microsoft/onnxruntime#21785
- 4: https://app.unpkg.com/onnxruntime-web@1.19.0/files/package.json
🌐 Web query:
registry.npmjs.org @ricky0123/vad-web 0.0.30 package.json
💡 Result:
The package @ricky0123/vad-web version 0.0.30 was published to the npm registry on November 21, 2025 [1][2][3]. It is a voice activity detection library designed for browser environments, utilizing onnxruntime-web [1][4][5]. While the full JSON contents are managed by the npm registry, the package structure at version 0.0.30 includes a dist directory containing the bundle (bundle.min.js, bundle.dev.js), various TypeScript definition files (.d.ts), and model-related JavaScript files [6][7]. You can access the package.json file specifically for version 0.0.30 directly through the npm registry's endpoint or via community tools that index package source code [6]. The package is licensed under the ISC license and depends on onnxruntime-web [1][2].
Citations:
- 1: https://registry.npmjs.org/%40ricky0123%2Fvad-web
- 2: https://www.npmjs.com/package/@ricky0123/vad-web
- 3: https://github.com/ricky0123/vad/releases/tag/%40ricky0123%2Fvad-web%2Fv0.0.30
- 4: https://npmx.dev/package/@ricky0123/vad-web/v/0.0.30
- 5: https://github.com/ricky0123/vad
- 6: https://npmx.dev/package-code/@ricky0123/vad-web/v/0.0.30
- 7: https://www.npmjs.com/package/@ricky0123/vad-web?activeTab=code
🌐 Web query:
cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.30/package.json
💡 Result:
The file located at cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.30/package.json is the package manifest for version 0.0.30 of the @ricky0123/vad-web npm package [1][2]. This package provides a voice activity detector (VAD) designed for browser environments [3][4][5]. Version 0.0.30 was published on November 21, 2025 [4]. The package.json file defines metadata for the project, including its dependencies (such as onnxruntime-web), entry points, and project configuration [3][4]. You can view the contents of this file directly through various CDN and package registry interfaces that host the package [1][6].
Citations:
- 1: https://npmx.dev/package-code/@ricky0123/vad-web/v/0.0.30
- 2: https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.30/
- 3: https://www.npmjs.com/package/@ricky0123/vad-web
- 4: https://registry.npmjs.org/%40ricky0123%2Fvad-web
- 5: https://docs.vad.ricky0123.com/user-guide/browser/
- 6: https://app.unpkg.com/@ricky0123/vad-web@0.0.30
🌐 Web query:
cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.30/package.json main module exports
💡 Result:
The @ricky0123/vad-web package, specifically version 0.0.30, is designed for browser-based voice activity detection [1][2]. Based on the provided file structure and documentation, the package provides its functionality via modules located in the dist directory [2][3]. While the specific raw content of the package.json file was not extracted, the documentation and project file structures confirm that the package is an ESM-compatible library [4]. In typical modern usage, the package provides named exports such as MicVAD, which users import directly from the package [5][6][7]. Developers generally access the library by importing from the package name, for example: import { MicVAD } from "@ricky0123/vad-web"; Internally, the package maps these exports to files within the dist folder, such as dist/index.js [2][3]. Because the package relies on browser-specific APIs and assets like Web Audio worklets and ONNX runtime files, bundling often requires configuring tools like Vite or Webpack to correctly serve the static files located in the dist directory (e.g., vad.worklet.bundle.min.js, silero_vad_v5.onnx, and WASM binaries) [8][9][10][7].
Citations:
- 1: https://www.npmjs.com/package/@ricky0123/vad-web
- 2: https://registry.npmjs.org/%40ricky0123%2Fvad-web
- 3: https://www.npmjs.com/package/@workadventure/noise-suppression
- 4: https://npmx.dev/package/@ricky0123/vad-web/v/0.0.30
- 5: https://docs.vad.ricky0123.com/user-guide/api/
- 6: Trying to get this bundled into a vite/SvelteKit app ricky0123/vad#34
- 7: Unable to run vad-web in VueJS ricky0123/vad#101
- 8: https://docs.vad.ricky0123.com/user-guide/browser/
- 9: https://github.com/Hyperwindmill/caretaker-cli/blob/main/docs/superpowers/plans/2026-07-25-voice-mode.md
- 10: If you use Vite, refer to my configuration. ricky0123/vad#128
🌐 Web query:
"https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.30/package.json" | sed -n '1,80p'
💡 Result:
The package @ricky0123/vad-web version 0.0.30 is a JavaScript library designed for performing Voice Activity Detection (VAD) directly in the web browser [1][2]. It allows developers to easily detect when a user is speaking by leveraging machine learning models (specifically using onnxruntime-web) [3][4]. The package is part of the ricky0123/vad project, which focuses on providing accurate, user-friendly voice activity detection [4][2]. Version 0.0.30 was published on November 21, 2025 [5][6]. Key updates in this release included fixes for custom audio context handling, a memory leak related to onnxruntime, and various API changes [6]. You can install it via npm or yarn: npm install @ricky0123/vad-web@0.0.30 [7] Usage typically involves importing MicVAD and initializing it to process microphone input [8]: import { MicVAD } from "@ricky0123/vad-web" const myvad = await MicVAD.new({ onSpeechEnd: (audio) => { // Process audio (Float32Array) }, }) myvad.start Documentation and further examples can be found at https://docs.vad.ricky0123.com/ [4].
Citations:
- 1: https://docs.vad.ricky0123.com/user-guide/browser/
- 2: https://www.vad.ricky0123.com/
- 3: https://www.npmjs.com/package/@ricky0123/vad-web
- 4: https://github.com/ricky0123/vad
- 5: https://registry.npmjs.org/%40ricky0123%2Fvad-web
- 6: https://github.com/ricky0123/vad/releases/tag/%40ricky0123%2Fvad-web%2Fv0.0.30
- 7: https://npmx.dev/package/@ricky0123/vad-web/v/0.0.30
- 8: https://docs.vad.ricky0123.com/user-guide/api/
🌐 Web query:
npmx.dev package-code @ricky0123/vad-web v0.0.30 package.json
💡 Result:
The package.json file for @ricky0123/vad-web version 0.0.30 can be accessed and viewed online via the npmx.dev service at the following URL: https://npmx.dev/package-code/@ricky0123/vad-web/v/0.0.30/package.json [1]. This file is also available on other package content delivery networks such as UNPKG: https://app.unpkg.com/@ricky0123/vad-web@0.0.30/files/package.json [2]. The @ricky0123/vad-web package, maintained by ricky0123, is a browser-based voice activity detector [3][4]. Version 0.0.30 was published on November 21, 2025 [5][6].
Citations:
- 1: https://npmx.dev/package-code/@ricky0123/vad-web/v/0.0.30
- 2: https://app.unpkg.com/@ricky0123/vad-web@0.0.30
- 3: https://npmx.dev/package/@ricky0123/vad-web/v/0.0.30
- 4: https://github.com/ricky0123/vad
- 5: https://registry.npmjs.org/%40ricky0123%2Fvad-web
- 6: https://github.com/ricky0123/vad/releases/tag/%40ricky0123%2Fvad-web%2Fv0.0.30
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== raw vad package manifests ==="
for version in 0.0.30; do
pkg='`@ricky0123/vad-web`'
echo "--- $pkg@$version package.json ---"
curl -fsSL "https://cdn.jsdelivr.net/npm/$pkg@$version/package.json" | sed -n '1,80p' || true
done
echo "=== raw onnxruntime-web package manifests ==="
for version in 1.19.0 1.19.2; do
pkg='onnxruntime-web'
echo "--- $pkg@$version package.json ---"
curl -fsSL "https://cdn.jsdelivr.net/npm/$pkg@$version/package.json" | sed -n '1,120p' || true
done
echo "=== resolve dist directories from manifests by package root ==="
python3 - <<'PY'
import json, urllib.request
packages = [("onnxruntime-web", "1.19.2"), ("`@ricky0123/vad-web`", "0.0.30")]
for p,v in packages:
url = "https://cdn.jsdelivr.net/npm/"+p+"@"+v+"/package.json"
pkg=j=json.load(urllib.request.urlopen(url))
print(f"{p}@{v} main={pkg.get('main')!r} module={pkg.get('module')!r} browser={pkg.get('browser')!r} exports={json.dumps(pkg.get('exports',''), sort_keys=True)[:200]!r}")
PYRepository: pikkujs/pikku
Length of output: 10911
Resolve assets from their real package asset directory.
distOf uses the package entry point directory, but onnxruntime-web resolves ESM to dist/ort.bundle.min.mjs, and @ricky0123/vad-web resolves to dist/index.js. bundle.min.js is at the package root, while the listed onnxruntime-web assets are at dist, so /voice-demo/lib/bundle.min.js can return 404 instead of a clear failure. Use package-specific resolve targets for each allow-list group.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@e2e/packages/functions/src/functions/voice-demo.function.ts` around lines 44
- 93, Update the asset resolution used with SERVABLE and distOf so each package
group resolves from the correct asset directory rather than assuming the package
entry-point directory. Use package-specific resolve targets: the voice assets
from the voice package, vad-web’s root-level bundle assets from its package
root, and onnxruntime-web’s listed runtime assets from its dist directory;
preserve the existing allow-list and clear failure behavior for unresolved
assets.
| /** | ||
| * Stop the agent talking. | ||
| * | ||
| * This is what `agentInterrupt()` wraps, written out because the inspector | ||
| * reads `func` statically and cannot see through a spread. `rpc.agent.interrupt` | ||
| * checks the run belongs to the caller before stopping it, and resolves | ||
| * `stopped: false` when there was nothing left to stop — racing a run that | ||
| * finishes on its own is the normal case here, not an error. | ||
| */ | ||
| export const voiceDemoInterrupt = pikkuSessionlessFunc({ | ||
| description: 'Interrupts the run the demo page is currently listening to', | ||
| input: VoiceDemoInterruptInput, | ||
| func: async (_services, { runId, reason }, { rpc }) => | ||
| rpc.agent.interrupt(runId, reason), | ||
| }) |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Inspect the agent interrupt RPC to see how it establishes run ownership.
set -euo pipefail
fd -t f 'ai-agent-interrupt.ts' packages -x sed -n '1,200p' {}
echo "=== interrupt registrations / ownership checks ==="
rg -nP -C 6 '\binterrupt\s*[:(]' packages/core/src --type=ts -g '!**/*.test.ts'
echo "=== session/userId use in interrupt paths ==="
rg -nP -C 4 '(userId|session|resourceId)' packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts 2>/dev/null || trueRepository: pikkujs/pikku
Length of output: 11655
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== ai-agent-interrupt implementation around ownership comments ==="
sed -n '180,260p' packages/core/src/wirings/ai-agent/ai-agent-interrupt.ts
echo "=== rpc-runner interrupt implementation ==="
sed -n '500,545p' packages/core/src/wirings/rpc/rpc-runner.ts
echo "=== function wiring declaration ==="
fd -t f 'voice-demo.function.ts|voice-demo.wirings.ts' e2e -x sh -c 'echo "--- $1"; sed -n "280,330p" "$1"' sh {}Repository: pikkujs/pikku
Length of output: 6388
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== interruptAIAgent implementation ==="
grep -n "export const interruptAIAgent\|interruptAIAgent" packages/core/src/wirings/ai-agent/*.ts
fd -t f 'ai-agent-interrupt.ts|ai-agent-*.ts' packages/core/src/wirings/ai-agent -x sh -c 'echo "--- $1"; rg -n -C 4 -C 12 "export const interruptAIIgent|function interruptAIIgent|const interruptAIIgent|interruptAIIgent" "$1"' sh {}
# exact spelling:
rg -n -C 8 "export const interruptAIAgent|function interruptAIAgent|const interruptAIAgent|interruptAIAgent" packages/core/src/wirings/ai-agent packages/core/src --type=ts
echo "=== voice-demo wiring full ==="
fd -t f 'voice-demo.wirings.ts' e2e -x sh -c 'echo "--- $1"; cat -n "$1"' sh {}Repository: pikkujs/pikku
Length of output: 1919
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== interruptAIAgent implementation ==="
sed -n '940,1045p' packages/core/src/wirings/ai-agent/ai-agent-stream.ts
echo "=== interruptAIAgent authorization test ==="
sed -n '682,845p' packages/core/src/wirings/ai-agent/ai-agent-interrupt.test.ts
echo "=== voice-demo wiring ==="
fd -t f 'voice-demo.wirings.ts' e2e -x sh -c 'echo "--- $1"; cat -n "$1"' sh {}Repository: pikkujs/pikku
Length of output: 10431
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== voice-demo function interrupt registration ==="
sed -n '1,180p' e2e/packages/functions/src/functions/voice-demo.function.ts
sed -n '300,450p' e2e/packages/functions/src/functions/voice-demo.function.ts
echo "=== session resolution helper ==="
rg -n -C 5 "export function |function resolveOwnerResourceId|resolveOwnerResourceId|assertResourceOwner" packages/core/src --type=ts
echo "=== all voice-demo run creation references ==="
rg -n -C 3 "runId|voiceDemo|interrupt" e2e/packages/functions/src/functions/voice-demo.function.ts e2e/packages/functions/src/wirings/voice-demo.wirings.tsRepository: pikkujs/pikku
Length of output: 50372
No ownership check exists on this route.
rpc.agent.interrupt calls interruptAIAgent, which compares the session principal to the run ownership and rejects missing sessions. Because this is a pikkuSessionlessFunc wired with auth: false, the demo interrupt cannot rely on ownership proof; either this route should establish the demo’s owner/principal, accept only owned runs during stream runtime, or correct the comment to not assert an absent check.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@e2e/packages/functions/src/functions/voice-demo.function.ts` around lines 304
- 318, The voiceDemoInterrupt route currently claims rpc.agent.interrupt
performs caller ownership validation despite being sessionless with auth
disabled. Update the route to establish and pass the demo owner/principal before
calling rpc.agent.interrupt, or restrict interruption to runs verified as owned
during stream runtime; otherwise remove the inaccurate ownership assertion from
the function comment.
| const { stopped, inFlightTools } = await post('/voice-demo/interrupt', { | ||
| runId, | ||
| reason: 'speech', | ||
| }) | ||
| if (stopped && inFlightTools.length > 0) { | ||
| // Named, not offered to undo — it has already happened. | ||
| say('note', 'still finishing: ' + inFlightTools.join(', ')) | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Guard inFlightTools before reading length.
The && short-circuit protects inFlightTools only when stopped is falsy. If the interrupt response omits inFlightTools while stopped is true, inFlightTools.length throws. The surrounding catch then shows the TypeError as a page note, which reads as an interrupt failure even though the interrupt succeeded.
🛡️ Proposed fix
- if (stopped && inFlightTools.length > 0) {
+ if (stopped && inFlightTools?.length > 0) {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const { stopped, inFlightTools } = await post('/voice-demo/interrupt', { | |
| runId, | |
| reason: 'speech', | |
| }) | |
| if (stopped && inFlightTools.length > 0) { | |
| // Named, not offered to undo — it has already happened. | |
| say('note', 'still finishing: ' + inFlightTools.join(', ')) | |
| } | |
| const { stopped, inFlightTools } = await post('/voice-demo/interrupt', { | |
| runId, | |
| reason: 'speech', | |
| }) | |
| if (stopped && inFlightTools?.length > 0) { | |
| // Named, not offered to undo — it has already happened. | |
| say('note', 'still finishing: ' + inFlightTools.join(', ')) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@e2e/packages/functions/src/functions/voice-demo.function.ts` around lines 748
- 755, Guard inFlightTools before accessing its length in the interrupt handling
logic after post('/voice-demo/interrupt'). Preserve the existing note behavior
when stopped is true and tools are present, while treating an omitted or
undefined inFlightTools value as an empty list.
| } catch (error) { | ||
| say('note', 'microphone unavailable: ' + error.message) | ||
| session = null | ||
| playback = null | ||
| $vad.disabled = false | ||
| setState('idle') |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Destroy the playback queue when the session fails to start.
startSession constructs AudioPlaybackQueue before it constructs VoiceSession. If the microphone prompt is denied, or session.start() throws for any other reason, this catch sets playback to null without calling destroy(). The AudioContext it holds is never closed. Browsers cap concurrent AudioContexts, so repeated denied attempts — the common case this branch exists for — eventually fail with an unrelated error.
🛡️ Proposed fix
} catch (error) {
say('note', 'microphone unavailable: ' + error.message)
+ void playback?.destroy()
+ void session?.destroy()
session = null
playback = null
$vad.disabled = false
setState('idle')
} finally {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| } catch (error) { | |
| say('note', 'microphone unavailable: ' + error.message) | |
| session = null | |
| playback = null | |
| $vad.disabled = false | |
| setState('idle') | |
| } catch (error) { | |
| say('note', 'microphone unavailable: ' + error.message) | |
| void playback?.destroy() | |
| void session?.destroy() | |
| session = null | |
| playback = null | |
| $vad.disabled = false | |
| setState('idle') |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@e2e/packages/functions/src/functions/voice-demo.function.ts` around lines 880
- 885, Update the startSession error path around the shown catch block to
destroy the existing AudioPlaybackQueue before clearing playback. Invoke
playback.destroy() when the queue was created, then set playback to null while
preserving the existing session reset and idle-state behavior.
| // Silence comes from the empty queue, not from a suspended context. Barge-in | ||
| // reaches here via `pause()`, and leaving the context suspended would mean | ||
| // the *next* reply decodes, queues and starts a source node that plays to | ||
| // nobody — the conversation goes quiet from the second turn onwards. | ||
| void this.resume() | ||
|
|
||
| return { text: heard.join(' '), complete } | ||
| } | ||
|
|
||
| /** Release the audio context. The queue is unusable afterwards. */ | ||
| async destroy(): Promise<void> { | ||
| this.interrupt() | ||
| await this.ctx.close() | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
destroy() can produce an unhandled promise rejection.
destroy() calls interrupt(), which starts this.resume() without awaiting it. destroy() then closes the context. If close() settles first, resume() rejects with InvalidStateError on a closed AudioContext. The void discards that rejection, so it surfaces as an unhandled rejection.
Attach a catch, or skip the resume when the context is closing.
🐛 Proposed fix
- void this.resume()
+ void this.resume().catch(() => {
+ // The context was closed underneath us (destroy); nothing to resume.
+ })📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // Silence comes from the empty queue, not from a suspended context. Barge-in | |
| // reaches here via `pause()`, and leaving the context suspended would mean | |
| // the *next* reply decodes, queues and starts a source node that plays to | |
| // nobody — the conversation goes quiet from the second turn onwards. | |
| void this.resume() | |
| return { text: heard.join(' '), complete } | |
| } | |
| /** Release the audio context. The queue is unusable afterwards. */ | |
| async destroy(): Promise<void> { | |
| this.interrupt() | |
| await this.ctx.close() | |
| } | |
| // Silence comes from the empty queue, not from a suspended context. Barge-in | |
| // reaches here via `pause()`, and leaving the context suspended would mean | |
| // the *next* reply decodes, queues and starts a source node that plays to | |
| // nobody — the conversation goes quiet from the second turn onwards. | |
| void this.resume().catch(() => { | |
| // The context was closed underneath us (destroy); nothing to resume. | |
| }) | |
| return { text: heard.join(' '), complete } | |
| } | |
| /** Release the audio context. The queue is unusable afterwards. */ | |
| async destroy(): Promise<void> { | |
| this.interrupt() | |
| await this.ctx.close() | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/voice-agents/src/audio-playback-queue.ts` around lines 140 - 153,
Update destroy() and the interrupt/resume flow so the asynchronous resume
triggered by interrupt() cannot become an unhandled rejection when ctx.close()
runs; either await and handle the resume before closing the AudioContext or
attach an explicit rejection handler, while preserving the queue’s existing
interruption behavior.
| try { | ||
| await vad.start() | ||
| } catch (error) { | ||
| handlers.onError?.( | ||
| error instanceof Error ? error : new Error(String(error)) | ||
| ) | ||
| } | ||
|
|
||
| return { | ||
| stop: () => { | ||
| if (stopped) return | ||
| stopped = true | ||
| void vad.destroy() | ||
| }, | ||
| } | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Catch vad.destroy() rejections.
stop() calls void vad.destroy() without a .catch(). If destroy() rejects — for example after a vad.start() failure at line 264 left the instance partially initialized — this becomes an unhandled promise rejection.
🔧 Proposed fix
return {
stop: () => {
if (stopped) return
stopped = true
- void vad.destroy()
+ vad.destroy().catch(() => {})
},
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| try { | |
| await vad.start() | |
| } catch (error) { | |
| handlers.onError?.( | |
| error instanceof Error ? error : new Error(String(error)) | |
| ) | |
| } | |
| return { | |
| stop: () => { | |
| if (stopped) return | |
| stopped = true | |
| void vad.destroy() | |
| }, | |
| } | |
| } | |
| try { | |
| await vad.start() | |
| } catch (error) { | |
| handlers.onError?.( | |
| error instanceof Error ? error : new Error(String(error)) | |
| ) | |
| } | |
| return { | |
| stop: () => { | |
| if (stopped) return | |
| stopped = true | |
| vad.destroy().catch(() => {}) | |
| }, | |
| } | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/voice-agents/src/speech-detector.ts` around lines 262 - 277, Update
the stop method returned by the speech detector setup to attach a rejection
handler to vad.destroy(). Route any destruction error through handlers.onError,
preserving the existing stopped guard and fire-and-forget behavior.
| const DENIALS = [ | ||
| 'do not', | ||
| 'dont', | ||
| 'never mind', | ||
| 'nevermind', | ||
| 'hold on', | ||
| 'wait', | ||
| 'stop', | ||
| 'cancel', | ||
| 'forget it', | ||
| 'negative', | ||
| 'denied', | ||
| 'deny', | ||
| 'no', | ||
| 'nope', | ||
| 'nah', | ||
| ] |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
interpretConsent reads "not okay" as consent.
normalise keeps the word "not". DENIALS has no bare not entry, and \bno\b does not match "not". So takePhrases finds the grant word and no denial. interpretConsent('not okay') returns 'granted'. The same applies to "not yes", "not fine", and "that's not okay". This grants a tool approval that the user refused.
Detect a negator before a grant phrase, or add not to the denial phrases.
🐛 Proposed fix
const DENIALS = [
'do not',
'dont',
+ 'not',
'never mind',Add covering cases to spoken-approval.test.ts:
test('does not read a negated grant as consent', () => {
assert.equal(interpretConsent('not okay'), 'denied')
assert.equal(interpretConsent('that is not fine'), 'denied')
})Also applies to: 181-192
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/voice-agents/src/spoken-approval.ts` around lines 131 - 147, Update
interpretConsent and its phrase matching around DENIALS/takePhrases so negated
grant phrases such as “not okay” and “that is not fine” return “denied” rather
than “granted”; detect “not” before a grant phrase or include it in the denial
handling, and add the specified covering assertions to spoken-approval.test.ts.
| /** Milliseconds of silence that end a turn. Defaults to 1200. */ | ||
| silenceDelay?: number | ||
| /** Analyser noise floor in dBFS. Defaults to -50. */ | ||
| minDecibels?: number |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Fix the documented default for silenceDelay.
The doc comment states 1200. captureNextTurn uses this.options.silenceDelay ?? 700 at Line 322. Align the doc with the code.
📝 Proposed doc fix
- /** Milliseconds of silence that end a turn. Defaults to 1200. */
+ /** Milliseconds of silence that end a turn. Defaults to 700. */
silenceDelay?: number📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| /** Milliseconds of silence that end a turn. Defaults to 1200. */ | |
| silenceDelay?: number | |
| /** Analyser noise floor in dBFS. Defaults to -50. */ | |
| minDecibels?: number | |
| /** Milliseconds of silence that end a turn. Defaults to 700. */ | |
| silenceDelay?: number | |
| /** Analyser noise floor in dBFS. Defaults to -50. */ | |
| minDecibels?: number |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/voice-agents/src/voice-session.ts` around lines 21 - 24, Update the
silenceDelay documentation in the voice session options to state the actual
default of 700 milliseconds, matching the fallback used by captureNextTurn;
leave the minDecibels documentation unchanged.
| async start(): Promise<void> { | ||
| if (this.listening) return | ||
| await this.acquire() | ||
| // Before `listening`, so the first turn is armed knowing which detector it | ||
| // has. Loading the model takes a second or two on a cold cache, and that | ||
| // wait belongs here rather than in the middle of the user's first sentence. | ||
| if (this.options.speech) await this.startSpeechDetector() | ||
| this.listening = true | ||
| this.captureNextTurn() | ||
| } | ||
|
|
||
| /** Stop waiting for turns but keep the microphone open, so resuming does not | ||
| * renegotiate the device. Use this between conversations, not {@link destroy}. */ | ||
| stop(): void { | ||
| this.listening = false | ||
| this.endTurn() | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
start() after stop() creates a second speech detector and leaks the first.
stop() clears listening and calls endTurn(), but it never calls this.stopSpeech. The detector stays armed. A later start() runs startSpeechDetector() again, because acquire() returns early and nothing guards on this.stopSpeech. The new stop function overwrites this.stopSpeech, so the first MicVAD instance can never be destroyed. It keeps running the model and keeps invoking onSpeechEnd, so each turn is emitted twice once listening is true again.
useVoiceConversation reaches this path directly: stop() then start() reuses the existing sessionRef.current and calls session.start().
Guard the detector startup so it is armed at most once per acquired stream.
🐛 Proposed fix
async start(): Promise<void> {
if (this.listening) return
await this.acquire()
// Before `listening`, so the first turn is armed knowing which detector it
// has. Loading the model takes a second or two on a cold cache, and that
// wait belongs here rather than in the middle of the user's first sentence.
- if (this.options.speech) await this.startSpeechDetector()
+ if (this.options.speech && !this.stopSpeech) {
+ await this.startSpeechDetector()
+ }
this.listening = true
this.captureNextTurn()
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| async start(): Promise<void> { | |
| if (this.listening) return | |
| await this.acquire() | |
| // Before `listening`, so the first turn is armed knowing which detector it | |
| // has. Loading the model takes a second or two on a cold cache, and that | |
| // wait belongs here rather than in the middle of the user's first sentence. | |
| if (this.options.speech) await this.startSpeechDetector() | |
| this.listening = true | |
| this.captureNextTurn() | |
| } | |
| /** Stop waiting for turns but keep the microphone open, so resuming does not | |
| * renegotiate the device. Use this between conversations, not {@link destroy}. */ | |
| stop(): void { | |
| this.listening = false | |
| this.endTurn() | |
| } | |
| async start(): Promise<void> { | |
| if (this.listening) return | |
| await this.acquire() | |
| // Before `listening`, so the first turn is armed knowing which detector it | |
| // has. Loading the model takes a second or two on a cold cache, and that | |
| // wait belongs here rather than in the middle of the user's first sentence. | |
| if (this.options.speech && !this.stopSpeech) { | |
| await this.startSpeechDetector() | |
| } | |
| this.listening = true | |
| this.captureNextTurn() | |
| } | |
| /** Stop waiting for turns but keep the microphone open, so resuming does not | |
| * renegotiate the device. Use this between conversations, not {`@link` destroy}. */ | |
| stop(): void { | |
| this.listening = false | |
| this.endTurn() | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/voice-agents/src/voice-session.ts` around lines 134 - 150, Ensure
the start() flow arms the speech detector at most once per acquired stream by
guarding startSpeechDetector() with the existing detector state, such as
this.stopSpeech. Preserve detector reuse across stop() followed by start(),
while still starting it when no detector has been created.
Spike: a Pikku agent you can hold a spoken conversation with, and a way to test that it actually did what it said.
Two halves, deliberately split so the OSS story stays clean: the API side lives in
core, the browser side lives in a new@pikku/voice-agentspackage, and neither knows about the other's runtime.What's here
core— cooperative interrupt. A verbal agent needs something a text one never did: the ability to be stopped mid-turn without losing what already happened.abortScopecarries a cancellation signal down through the function runner, andbeginChanges()gives an agent a checkpoint it can be interrupted at rather than anywhere.interruptedjoins the AI run status enum, kept distinct fromfailed— the run was stopped on purpose, usually because the user talked over it, so it shouldn't surface in error reporting.@pikku/voice-agents(new). Mic capture, speech and silence detection, playback queueing, spoken approvals.@pikku/ai-deepinfra(new). STT/TTS provider. The voice path moved off Whisper once its invented filler turned out not to be filterable by confidence — it will produce a confident "Thank you." from silence.cli— two fixes that meantactor.conversehad never once worked underpikku scenario run.Scenario steps execute in the CLI process, which stubbed its singleton services to three entries, so every conversing scenario died on
AIProviderNotConfiguredbefore saying anything. It now builds a runner the same waypikku devdoes, and only when the project declares agents, so agent-less projects still need no AI env.The second is worse because it is silent.
supportsStructuredOutputsdefaults tofalse, which dropsoutputSchema, degrades the call to barejson_object, and returns well-formed JSON with whatever keys the model felt like. Nothing errors; the caller readsundefinedoff every field it asked for and behaves as though the model said nothing.Also adds
scenarios.actorModel, so the model the actor thinks with is configurable and explicitly not the model under test.The testing idea
A voice agent once replied "Okay — marking get lunch done." having called no tool at all, and on another run reached for
deleteTodo. Both of those pass a test that only reads the actor's verdict, because the actor is judging the conversation, and the conversation was fine.So the assertions go to the store instead: the todo exists, it is completed, exactly one of it exists, and the todo nobody mentioned is untouched.
Verified negatively — with
completeTodoremoved from the agent, the run produces:The actor's verdict passes while the store shows nothing happened. That's the case a verdict-only test ships green.
Note what is not asserted: which tools ran. The user asked for a todo to be done, not for
completeTodoto be invoked. Whether the agent's sentence was honest about how it got there is a grade on the run rather than a check on the world, and belongs to the judge in #719.Actor model
gpt-4.1-mini, chosen against a bench of the actor's four real behaviours — return each schema, and end the conversation when the goal is met. Every cheaper candidate failed the last one:Mistral-Small-3.2-24Bscored 0/4 on "say you're done", so it burned the full 12-turn budget and took 320s instead of 33s. Per-token savings are an illusion there, because each wasted turn also bills the agent's model.Verification
@pikku/core— 1591 pass, 0 fail@pikku/ai-vercel33,@pikku/ai-deepinfra22,@pikku/voice-agents28 — all passingmain; three conflicts resolved by hand (voice-output.tsfor the newemit/signalparams,scenario.tsalongside the newsetRunSurface, and thefunc:→default:step rename), then the full suite re-runOne intentional behaviour change — please look at this
donenow goes through the stream middleware rather than straight at the channel, and is awaited:It is the only signal a stream hook gets that the reply is over, and hooks that buffer need it —
voiceOutputspeaks a trailing fragment that never reached a full stop, and waits on audio it has already paid to synthesize. Sent raw, that work is thrown away by theclose()on the next line.This changes observable middleware ordering, so
verifiers/middleware-and-permissionsis updated:modifyOutputStreamnow fires once more, aftermodifyOutput. That ordering is deliberate —modifyOutputis about the completed reply,doneis about the stream carrying it — but it is a spec change and anyone with amodifyOutputStreamhook will now see one extra event. Shout if that trade is wrong; the alternative is a separate flush hook that only buffering middleware implements.The pre-push verifier suite is what caught this, not me.
Reviewer notes
e2e/pikku.config.jsonshows some unrelated array-formatting churn. That'slint-staged's prettier reformatting a file whose committed copy onmainwas stale — not a hand edit.tscerror remains inpackages/cli/src/services.ts(strictMeta), untouched by this branch and present onmain.packages/voice-agentshas no README beyond a stub and the browser side has had far less adversarial testing than the core side.🤖 Generated with Claude Code
Summary by CodeRabbit