feat(telemetry): trace the end-of-turn wait and the turn handoff by davidzhao · Pull Request #7130 · livekit/agents

davidzhao · 2026-09-05T07:55:53Z

What

eou_detection only covered the turn-detector inference, so the endpointing delay itself (often seconds) showed up as an empty gap between user_turn and agent_turn. This makes the wait, the user hook, and the speech queue visible.

How

eou_wait span, one per user turn, child of user_turn:

back-dated to last_speaking_time, so the bar starts where the user stopped talking;
ends on the turn decision with lk.eou.outcome (committed / user_resumed / dropped), lk.eou.wait_duration, lk.eou.rearm_count, the final lk.eou.endpointing_delay, and the last lk.eou.source;
a later trigger for the same turn (late STT final, another VAD end of speech) re-arms the wait and records a rearmed event instead of starting a new span, so the bar stays whole and late transcripts are readable off it;
a rejected commit (min_words, realtime backchannel) records a lk.eou.not_committed_count and keeps waiting; resumed speech ends the span at the VAD/STT speech start; teardown drops it.
resumed speech ends the span at the VAD/STT speech start, kept as a user_resumed event; when that start is reported after the fact and a detection or re-arm was recorded in between, the bar runs to the last of those so it always contains its children (an eou_detection still running is closed with a superseded event). A turn-detection mode change to or from manual drops the pending wait.

eou_detection keeps its name and now nests under eou_wait; the span names and the lk.eou.* attribute keys stay on the same vocabulary, so nothing on the cloud side needs to change.

on_user_turn_completed span around the user hook that gates the reply, nested under the user_turn it completes: recognition hands the open user_turn span to the activity when it schedules the hook, and the activity ends it after the hook (or when the turn is skipped), so the turn's duration covers the wait for the hook. Exceptions are recorded redaction-aware, honouring record={"redaction": True} on the session as well as the job's flag; StopResponse is an event.

lk.speech.queue_wait on agent_turn: seconds between scheduling the speech and its first generation authorization.

user_turn
├─ user_speaking
└─ eou_wait                 ← new, 2.5s, rearmed ×1
   └─ eou_detection         ← now nested under the wait
on_user_turn_completed      ← new
agent_turn  (lk.speech.queue_wait=0.01)
├─ llm_node
└─ tts_node

Tests

tests/test_eou_wait_span.py drives _run_eou_detection with real spans: commit (parent, back-dated start, attributes), re-arm keeps one span, resumed speech ends at speech start and leaves user_turn open, teardown drops, not_committed keeps waiting, eou_detection nests under the wait and stays inside it when the user resumes mid-inference, a mode change drops the wait; plus a full fake session asserting eou_wait, the hook span, the queue-wait attribute, and a session-only redaction keeping a hook exception's message out of the trace.

Stacked on #7128.

🤖 Generated with Claude Code

Counts, not events. lk.eou.not_committed_count on eou_wait (turn decisions the detector rejected before the wait ended) replaces the not_committed event; lk.eou.resume_count on user_turn counts the waits the user cut short by speaking again.
Stages on the reply. The reply's agent_turn carries lk.end_of_turn_delay, lk.transcription_delay and lk.on_user_turn_completed_delay next to lk.e2e_latency, so a per-turn breakdown reads off one span.
A user_turn created lazily from an STT final (no VAD opened it) starts at the earliest anchor known, so its back-dated eou_wait child never starts before it.

jmcclanahan

Couple QoL things for downstream, otherwise lgtm

jmcclanahan · 2026-09-09T01:09:08Z

                    self._turn_detector_flushed = True

+            elif eou_wait_span.is_recording():
+                eou_wait_span.add_event("not_committed")


Could this also bump a counter that lands as lk.eou.not_committed_count next to lk.eou.rearm_count? Rate of rejected turn decisions is a fleet-level tuning signal, and events need the events array scanned, while an attribute is a cheap read.

`eou_detection` only covered the turn-detector inference, so the endpointing delay itself (often seconds) showed up as an empty gap between the user turn and the agent turn. - New `eot_wait` span, one per user turn, child of `user_turn`. It is back-dated to `last_speaking_time` so the bar starts where the user stopped talking, and ends on the turn decision with `lk.eou.outcome` = committed | user_resumed | dropped, `lk.eou.wait_duration`, `lk.eou.rearm_count`, the final `lk.eou.endpointing_delay`, and the last `lk.eou.source`. A later trigger for the same turn (late STT final, another VAD end of speech) re-arms the wait and records a `rearmed` event instead of starting a new span, so the bar stays whole; a rejected commit (min_words, realtime backchannel) records `not_committed` and keeps waiting. Resumed speech ends it at the VAD/STT speech start; teardown drops it. - `eou_detection` is renamed `eot_detection` and nests under `eot_wait`. The `lk.eou.*` attribute keys are unchanged. - New `on_user_turn_completed` span around the user hook that gates the reply, parented to the session root; exceptions are recorded (redaction-aware) and StopResponse is an event. - `agent_turn` gains `lk.speech.queue_wait`: seconds between scheduling the speech and its first generation authorization, recorded at all four authorization sites. Tests: tests/test_eot_wait_span.py drives `_run_eou_detection` with real spans (commit, rearm, user resumed, dropped, not committed, detection nesting) and a full fake session for the hook span and queue wait. Existing AudioRecognition test harnesses gain the new state fields. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

tests/test_realtime_reply_chat_ctx.py drives _realtime_reply_task with a SimpleNamespace standing in for the activity; a method call on self there raised AttributeError inside the task and the test then waited forever for a reply future, hanging the unit gate. Reply tasks must not depend on telemetry helpers living on the activity, so the helper is a module-level function. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

lk.eou.wait_duration and the span length came from the same float seconds but were truncated to nanoseconds separately, so they could differ by a nanosecond. Keep the start in nanoseconds and compute the attribute from the same two integers the span is bounded by; the test asserts exact equality. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…n is eou_wait A partial move to "eot" left a span named eot_detection carrying lk.eou.* attributes. Renaming the keys too would need a cloud-side migration that is not worth it right now, so both spans stay on the vocabulary the attribute keys already use. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…hanges, honour session redaction Review follow-ups on #7130: - a resumed-speech end time reported after the fact could land before an eou_detection that was still running or a re-arm recorded meanwhile; the wait now ends no earlier than what it contains, closes a running detection with a superseded event, and keeps the real resume time as a user_resumed event. - switching turn detection to or from manual cancelled the endpointing task but left the wait open; it is dropped with the decision. - on_user_turn_completed exceptions are redacted when redaction is enabled for the session alone, not only via the job flag. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ts under it The hook gates the reply, so the turn is not over until it has run. Recognition hands the open user_turn span to the activity when it schedules the hook; the activity parents the hook span to it and ends it after the hook (or when the turn is skipped), just before any reply is generated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…sk; rename to _end_user_turn_span The explicit call after the hook duplicated the task's finally. The NOTE about AgentActivity's visibility goes back above the class it describes. Comments trimmed to what a reader of the code needs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

_realtime_reply_task recorded it before _realtime_generation_task had opened the turn's span, so the attribute landed on whatever span was current. The generation task records it; the helper refuses to fall back to the current span. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Set only after detection, it was lost on a wait that resumed speech had already ended. The delay in force is stamped at creation and re-arm; a prediction that raises it overwrites while the span is still open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…an, lazy user_turn starts at its anchor - lk.eou.not_committed_count on eou_wait replaces the not_committed event; lk.eou.resume_count on user_turn counts the waits the user cut short - the reply's agent_turn carries lk.end_of_turn_delay, lk.transcription_delay and lk.on_user_turn_completed_delay next to lk.e2e_latency - a user_turn created from an STT final with no VAD start starts at the earliest known anchor, so the back-dated eou_wait child fits inside it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

davidzhao force-pushed the dz/telemetry-eot-wait branch from 71e137c to f52117e Compare September 5, 2026 20:07

davidzhao mentioned this pull request Sep 5, 2026

feat(telemetry): trace dispatch, startup, and shutdown #7131

Merged

davidzhao force-pushed the dz/telemetry-eot-wait branch 2 times, most recently from e5d040a to dfdc981 Compare September 5, 2026 21:00

davidzhao force-pushed the dz/telemetry-eot-wait branch 2 times, most recently from 0126c90 to c248a86 Compare September 5, 2026 21:10

davidzhao force-pushed the dz/telemetry-eot-wait branch 3 times, most recently from bae9049 to 3fc7868 Compare September 6, 2026 01:09

davidzhao marked this pull request as ready for review September 6, 2026 01:40

davidzhao requested a review from a team as a code owner September 6, 2026 01:41

This comment was marked as resolved.

Sign in to view

davidzhao force-pushed the dz/telemetry-eot-wait branch from f1b1d9f to 1348da7 Compare September 6, 2026 05:23

This comment was marked as resolved.

Sign in to view

davidzhao force-pushed the dz/telemetry-eot-wait branch 6 times, most recently from ea80860 to cc498be Compare September 6, 2026 08:09

davidzhao force-pushed the dz/telemetry-eot-wait branch 2 times, most recently from 45988c2 to bfa49c1 Compare September 6, 2026 16:48

davidzhao force-pushed the dz/telemetry-eot-wait branch from bfa49c1 to cf9e01f Compare September 6, 2026 17:13

davidzhao force-pushed the dz/telemetry-eot-wait branch from cf9e01f to 05c7265 Compare September 6, 2026 21:10

This comment was marked as outdated.

Sign in to view

jmcclanahan approved these changes Sep 9, 2026

View reviewed changes

davidzhao force-pushed the dz/telemetry-eot-wait branch from b02f1bd to 74fc258 Compare September 9, 2026 05:51

davidzhao force-pushed the dz/telemetry-eot-wait branch from 74fc258 to cad53f6 Compare September 9, 2026 05:55

davidzhao force-pushed the dz/telemetry-eot-wait branch 3 times, most recently from 6e8f532 to 327e89d Compare September 9, 2026 07:37

Base automatically changed from dz/telemetry-loop-monitor to main September 9, 2026 07:41

davidzhao force-pushed the dz/telemetry-eot-wait branch 2 times, most recently from 82d347a to a176fad Compare September 9, 2026 07:43

chenghao-mou approved these changes Sep 10, 2026

View reviewed changes

davidzhao and others added 12 commits September 11, 2026 15:01

voice: stamp the superseded event with the detection span's end time

b6add02

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

tests(eou_wait): widen the re-arm test's margin for slow runners

831f90c

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

davidzhao force-pushed the dz/telemetry-eot-wait branch from a8af38e to 796e9b4 Compare September 11, 2026 22:06

davidzhao merged commit bf2e7f7 into main Sep 12, 2026
23 checks passed

davidzhao deleted the dz/telemetry-eot-wait branch September 12, 2026 06:35

davidzhao mentioned this pull request Sep 15, 2026

feat(telemetry): trace the end-of-turn wait and the turn handoff livekit/agents-js#2496

Merged

Navigation Menu

Repository navigation

feat(telemetry): trace the end-of-turn wait and the turn handoff - #7130

feat(telemetry): trace the end-of-turn wait and the turn handoff#7130
davidzhao merged 12 commits into
mainfrom
dz/telemetry-eot-wait

davidzhao commented Sep 5, 2026 •

edited

Loading

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as outdated.

Uh oh!

jmcclanahan left a comment

Uh oh!

Uh oh!

jmcclanahan Sep 9, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

davidzhao commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Tests

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as outdated.

Uh oh!

jmcclanahan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jmcclanahan Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

davidzhao commented Sep 5, 2026 •

edited

Loading