Skip to content

plugins/bithuman: let a session name the second-generation models - #7366

Merged
longcw merged 5 commits into
livekit:mainfrom
sgu-bithuman:bithuman-model-v2
Sep 22, 2026
Merged

longcw merged 5 commits into
livekit:mainfrom
sgu-bithuman:bithuman-model-v2

Conversation

@sgu-bithuman

@sgu-bithuman sgu-bithuman commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

AvatarSession(model=...) accepts only "expression" and "essence", and the request it sends names no model at all β€” just mode: gpu|cpu, which can distinguish exactly two engines. Which model a second-generation avatar is served as therefore depends on a server-side default rather than on what the caller asked for, and the caller has no way to say.

Measured against BitHuman's live API

One account, and an avatar prepared for only one model (supported_models: ["expression-1"]). The payload on the wire is the one this plugin's own _send_json_request builds β€” only the transport wrapper was replaced so the response the plugin discards could be read.

payload result
1.8.2, model="essence" no model field, mode=cpu HTTP 200 β€” session started and served as expression-1
this PR, model="expression-2" model="expression-2", mode=gpu HTTP 409 β€” "agent …'s expression-2 model hasn't been generated yet β€” add it with POST /v1/agent/{code}/models, then retry"

A session started as a model the caller did not ask for, and nothing said so. The new refusal arrives before a renderer launches or a credit is spent.

On an avatar that is prepared, the same call is answered {"model": "expression-2", "avatar_session_started": true} where 1.8.2 was answered "expression-1". model="essence-2" reaches the Essence 2 launcher, which names the bundle it wants for that identity β€” so the string routes end to end.

Backward compatibility

The two first-generation names are unchanged and keep resolving to the first-generation engines: driven against the same API, "expression" and "essence" produce payloads identical to 1.8.2 apart from the added field, and the same responses. mode is still sent, unchanged, for servers that predate model; the server prefers the explicit field.

The gpu/cpu split and the custom-endpoint request format now read one property, _is_expression_model, so the two sites cannot disagree about which family a session is β€” dispatching one way and billing the other is the failure that shape invites.

Patch bump, 1.8.2 β†’ 1.8.3, per AGENTS.md.

πŸ€– Generated with Claude Code

https://claude.ai/code/session_012vTGmNLyJSwj28bfFk5gG2


Coverage after review (2026-09-21)

Two follow-ups from the Devin review, and one gap it surfaced by making me re-read the whole path.

1. An explicit NOT_GIVEN no longer serializes as a model name. NotGiven.__repr__ is "NOT_GIVEN" and the class defines no __str__, so str(NOT_GIVEN) is that literal string. Normalised once at the assignment rather than at the reported read site, because there are two readers β€” _is_expression_model and the model field β€” and that property's docstring requires them to agree. Patching only the JSON field would have left the agreement resting on a coincidence.

2. The self-hosted path now names the model too. The first commit taught only the cloud JSON request. A custom api_url routes to multipart/form-data, which sent no model at all β€” so a self-hosted worker could not distinguish expression from expression-2, two engines that render differently and meter on different lines.

Verified additive rather than assumed: /launch reads the form into a dict and picks known keys, so an unrecognised field is ignored. Measured against a running expression-2 worker β€” a form with no model, one with model=expression-2, and one with a deliberately garbage field all return the identical 400 for the missing required fields. No 422, no parse failure, so no existing deployment changes behaviour.

Where the model reaches the renderer

path transport how the model is chosen
local (model_path) in-process read from the model file; the runtime routes expression-2 to its own streaming runtime and derives the billing line from the engine id
self-hosted, expression family multipart/form-data now carries model
self-hosted, essence family JSON carries model
cloud API JSON carries model

Local mode was checked against the shipped bithuman 2.11.6 wheel rather than inferred from the signature: AsyncBithuman.create() takes no model argument and its billing_type default names essence, but that default is not what a second-generation engine ends up using β€” the loader reads the engine from the model header and calls billing_type_for_engine().

`AvatarSession(model=...)` accepted only "expression" and "essence", and the
request it sent named no model at all β€” just `mode: gpu|cpu`, which can
distinguish exactly two engines. Which model a second-generation avatar was
served as therefore depended on a server-side default rather than on what the
caller asked for, and the caller had no way to say.

Measured against BitHuman's live API with one account, on an avatar prepared for
only one model (`supported_models: ["expression-1"]`), the payload built by this
plugin's own `_send_json_request`:

  1.8.2   model="essence"       sends no model, mode=cpu
          -> HTTP 200, session STARTED and served as expression-1

A session started as a model the caller did not ask for, and nothing said so.

  this    model="expression-2"  sends model="expression-2", mode=gpu
          -> HTTP 409, "agent ...'s expression-2 model hasn't been generated
             yet - add it with POST /v1/agent/{code}/models ..., then retry"

...before a renderer launches or a credit is spent. On an avatar that IS
prepared, the same call is answered `{"model": "expression-2",
"avatar_session_started": true}` where 1.8.2 was answered "expression-1".

  model="essence-2" reaches the Essence 2 launcher, which names the bundle it
  wants for that identity - so the string routes end to end.

The two first-generation names are unchanged and keep resolving to the
first-generation engines: driven against the same API, "expression" and
"essence" produce byte-identical payloads to 1.8.2 apart from the added field,
and the same responses.

`mode` is still sent, unchanged, for servers that predate `model`; the server
prefers the explicit field. The gpu/cpu split and the custom-endpoint request
format now read one property, `_is_expression_model`, so the two sites cannot
disagree about which family a session is - dispatching one way and billing the
other is the failure that shape invites.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vTGmNLyJSwj28bfFk5gG2
@sgu-bithuman
sgu-bithuman requested a review from a team as a code owner September 20, 2026 16:28
devin-ai-integration[bot]

This comment was marked as resolved.

sgu-bithuman and others added 3 commits September 21, 2026 06:11
Devin review, and it is right: `NotGivenOr[BitHumanModel] = "essence"` lets a
caller pass NOT_GIVEN explicitly, and code forwarding an optional parameter does
that routinely. NotGiven defines `__repr__` as "NOT_GIVEN" and no `__str__`, so
`str(NOT_GIVEN)` is the literal string "NOT_GIVEN" (measured against the real
class in livekit/agents/types.py, not assumed). It would have been sent as the
model name and refused by the server, on a call the caller meant as "the
default".

Fixed at the ASSIGNMENT rather than at the reported read site, because there are
two readers and the reported one is not the only one at risk:

  self._model = model                              # __init__
  str(self._model).startswith("expression")        # _is_expression_model
  "model": str(self._model)                        # the session request

With NOT_GIVEN the second yields False and the session is dispatched "cpu". That
happens to agree with the essence default, so it is benign today β€” purely by
luck, and that property's own docstring says the two readers "must give the same
answer in both, or a session is dispatched one way and billed the other".
Patching only the JSON field would have left that agreement resting on a
coincidence. Normalising once makes both correct by construction.

Verified across every input the signature admits β€” NOT_GIVEN, essence,
essence-2, expression, expression-2 β€” that the model sent is always one of the
four valid names and that the dispatch mode matches it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134RFpn6JSLqWYJ88rJ9ohB
The first commit taught the CLOUD request to name the model. That left the
SELF-HOSTED path β€” a custom `api_url`, which routes to multipart/form-data β€”
still sending only livekit_url, livekit_token, room_name, async_mode,
avatar_image/avatar_image_url and avatar_id. No model. So a self-hosted worker
could not tell `expression` from `expression-2`: two engines that render
differently and meter on different lines, chosen by a server-side default rather
than by what the caller asked for. That is the same defect the JSON change
fixes, in the transport the change did not touch.

Adding it is additive for every existing deployment, verified against the live
worker rather than assumed: /launch reads the form into a dict and picks the
keys it knows, so an unrecognised field is ignored. Measured on the running
expression-2 worker β€” a form with no `model`, one with `model=expression-2`, and
one with a deliberately garbage field all return the identical 400 for the
missing required fields. No 422, no parse failure.

With this the model reaches the renderer on every path the plugin offers:

  local        engine read from the model file itself; the runtime routes
               expression-2 to Expression2StreamingRuntime and derives the
               billing line via billing_type_for_engine (verified in the
               shipped bithuman 2.11.6 wheel, not assumed from the signature β€”
               `create()` has no `model` argument and its billing_type default
               names essence, but that default is not what a v2 engine uses)
  self-hosted  FormData (expression family) / JSON (essence family) β€” both now
               carry `model`
  cloud API    JSON β€” carries `model`

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134RFpn6JSLqWYJ88rJ9ohB
Caught reviewing my own backward compatibility before asking for approval, and
it would have broken existing users.

The parameter defaulted to the literal "essence", and the two commits before this
one sent `model` on EVERY session. So a caller who had never heard of this
parameter β€” the whole installed base β€” would suddenly have been sending
`model: "essence"`. Naming a model is a PIN, not a hint: the API answers 400 and
names what the avatar can be served as if it was not prepared for that model. An
avatar prepared only as essence-2 would have gone from working to refused, on an
upgrade that touched none of its code.

So `model` now defaults to NOT_GIVEN and is sent only when the caller names one:

  unset            nothing is sent; BitHuman resolves the avatar exactly as it
                   did before this parameter existed
  named            sent, on both the cloud JSON path and the self-hosted
                   form-data path

Verified as equivalence, not by inspection: for a caller who names no model the
request body is byte-identical to pre-PR in both branches (`{"mode": "cpu"}` /
`{"mode": "gpu"}` with avatar_image), and the sentinel is never stringified, so
Devin's "NOT_GIVEN" can no longer be produced by any path at all β€” the earlier
normalise-to-"essence" fix is removed because it was the thing creating the
regression.

`_is_expression_model` gains the same guard, so the mode field and the model
field still agree on every input.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134RFpn6JSLqWYJ88rJ9ohB
devin-ai-integration[bot]

This comment was marked as resolved.

CI type-check, both 3.10 and 3.13:

    "AvatarSession" has no attribute "_model"; maybe "_mode"?   x5

The previous commit rewrote the block that assigns `self._model` in order to
replace the explanation above it. The replacement kept the comment and dropped
the line it was explaining, so `self._model` was never assigned at all β€” an
AttributeError on the first session, not merely a typing complaint.

Two things did not catch it, and both are worth naming. My local mypy run could
not resolve the base class in this environment, so the attribute error was
invisible among ten pre-existing import-resolution errors that are identical
before and after the change. And the behaviour matrix I ran to "prove"
equivalence modelled the LOGIC in a separate script rather than exercising the
module β€” so it happily proved a property of code that was no longer there. CI,
which installs the package properly, found it in one run.

Added a structural check while fixing it: every `self._X` that is read must also
be assigned somewhere in the class. It reports `_model` missing on the broken
version and clean on this one (`_room` is inherited and shows in the green
control too, so it is not a new finding).

Behaviour is unchanged from the intent of the previous commit: `model` defaults
to NOT_GIVEN and is sent only when the caller names one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134RFpn6JSLqWYJ88rJ9ohB
@sgu-bithuman

Copy link
Copy Markdown
Contributor Author

Summary for a reviewer, since this picked up a few commits.

What it does. BitHuman has two model families in two generations β€” expression/essence (first) and expression-2/essence-2 (current). The plugin had no way to say which one a session should use, so a second-generation avatar was resolved by a server-side default rather than by the caller. This lets a session name one.

Blast radius. Two files, both inside livekit-plugins-bithuman: avatar.py and version.py. No shared livekit-agents code, no new dependency, no public API removed.

Backward compatibility β€” the part worth checking. model defaults to NOT_GIVEN and is sent only when the caller names one. For a session that does not use the parameter, the outgoing request is byte-identical to before this PR ({"mode": "cpu"}, or "gpu" with avatar_image), and nothing new is transmitted.

That is deliberate and it is a correction I made to my own earlier commits in this PR. They defaulted the parameter to "essence" and sent it on every session. Naming a model is a pin, not a hint β€” the API answers 400 and names what the avatar can be served as if it was not prepared for that model β€” so an existing user whose avatar is prepared only as essence-2 would have gone from working to refused, on an upgrade that touched none of their code. Making it opt-in removes that entirely.

Review feedback. The one Devin finding (an explicit NOT_GIVEN serializing as the string "NOT_GIVEN") is resolved. I verified the premise against NotGiven in livekit/agents/types.py β€” __repr__ returns "NOT_GIVEN" and there is no __str__ β€” and fixed it at the assignment rather than the reported line, because two readers consume self._model and the other one agreed with the default only by coincidence. The sentinel is now never stringified on any path.

Verification.

  • NOT_GIVEN, essence, essence-2, expression, expression-2 each produce a valid request; the no-model case is identical to pre-PR in both the avatar_image and avatar_id branches.
  • The self-hosted (multipart/form-data) path also carries model now β€” it previously sent none, so a self-hosted worker could not distinguish expression from expression-2. Confirmed additive against a running worker: forms with no model, with model=expression-2, and with a deliberately unknown field all return the same 400 for missing required fields, so no existing deployment changes behaviour.
  • Local mode needs no change: the runtime reads the engine from the model file itself.

CI. ruff, type-check (3.10 and 3.13) and blockguard-tests (ubuntu/macos/windows) all green on the head commit.

Happy to squash the history if you'd prefer a single commit. Thanks for taking a look.

@longcw
longcw merged commit 014b448 into livekit:main Sep 22, 2026
16 checks passed
sgu-bithuman pushed a commit to bithuman-product/public-docs that referenced this pull request Sep 22, 2026
Three pages told a developer less than the product does. Each was written when
it was true and never re-read against the thing it describes.

1. /sdk, Essence 2 / Android β€” "the step-by-step is not on the Android page
   yet", and it pointed at the CLI, Python or the cloud API instead. Measured
   against sdk/android.md on origin/main: that page carries `## Install ###
   Essence 2` (L91) with the dependency line, `## Minimal code ### Essence 2`
   (L175) with the imports and the render loop, `## Get a model`, `##
   Authentication`, and it links a whole worked app. The hub card was the only
   thing standing between a reader and all of it.

2. /sdk, Essence 2 / Web and /sdk/web β€” both described the downloadable
   benchmark package (one built-in identity, cannot render your own agent) as
   if that were the whole of Essence 2 in a browser, and /sdk/web listed
   `?render=local` for Expression 2 only. It is not model-specific:
   apps/agent-ui/src/lib/local-render/family.ts dispatches essence1, essence2
   and expression2 to in-browser renderers and refuses only expression1, and
   sdk/performance.md's own Web / Essence 2 cell is measured on
   `<CODE>?render=local`. Measured anonymously on 2026-09-22, the public asset
   mirror answers HTTP 200 for 12 of 12 live Essence 2 identities and 11 of
   them publish the faster renderer's files (a bogus code answers HTTP 400, so
   a 200 means something). `https://www.bithuman.ai/A21SKT4314?render=local`
   redirects to the viewer with a session token whose payload reads
   `"model":"essence-2","anonymousEnabled":true`; a bogus code redirects to
   /not-found and answers 404. Both pages now say `?render=local` works for
   all three models that have an in-browser renderer, and Troubleshooting
   gains the two outcomes a reader actually meets: the page rewriting the URL
   to `?render=cloud` when an identity has no in-browser build, and the
   refusal message the renderer prints. No frame rate is repeated here β€” the
   performance page stays the one writer.

3. /sdk/livekit β€” three claims re-measured today:
   * "open upstream as livekit/agents#7366 and is not released yet" β€”
     #7366 MERGED 2026-09-22T02:23:29Z. Still unreleased (PyPI's newest
     livekit-plugins-bithuman is 1.8.2, 2026-09-15T18:16:50), so the page now
     says merged-and-unreleased rather than open. #7280 likewise merged
     2026-09-15T22:08:47Z, hours after 1.8.2 was cut.
   * the Swift package pinned `.upToNextMajor(from: "2.14.0")` while
     livekit/client-sdk-swift's newest release is 2.17.0 (2026-09-14) β€” three
     minors stale, and graded by nothing here: check-versions-current's V6
     tapPins only matches bithuman-product/homebrew-bithuman URLs.
   * "the local-essence and cloud-essence examples ship with this applied" β€”
     cloud-essence ships no tuned_publish.py and its agent.py names no publish
     option at all. It does not need one: it passes an `avatar_id`, so a
     bitHuman worker publishes the track. Only local-essence imports it.
   And the install line pins nothing, so the page now says which release it
   describes and how to check what you got.

Everything the page claims about the install was re-run in throwaway venvs
today: `pip install livekit-plugins-bithuman pillow` on 3.12 installs
1.8.2 + bithuman 2.11.6 + pillow 12.3.0; without pillow the first import dies
on `ModuleNotFoundError: No module named 'PIL'`; on 3.10 and on 3.14 the
marker is false, no wheel installs and the import dies on `No module named
'cv2'`. `AvatarSession.__init__` still declares
`model: Literal['expression','essence'] = 'essence'`, and the documented
monkey-patch still matches AvatarRunner._publish_track line for line, with
every private attribute it touches set in __init__. "There is no npm package"
also still holds: the registry's search for `bithuman` returns 0 results and
six candidate names answer 404, with livekit-client returning 200 as the
control.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134RFpn6JSLqWYJ88rJ9ohB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants