Repository navigation
plugins/bithuman: let a session name the second-generation models - #7366
Conversation
`AvatarSession(model=...)` accepted only "expression" and "essence", and the
request it sent named no model at all β just `mode: gpu|cpu`, which can
distinguish exactly two engines. Which model a second-generation avatar was
served as therefore depended on a server-side default rather than on what the
caller asked for, and the caller had no way to say.
Measured against BitHuman's live API with one account, on an avatar prepared for
only one model (`supported_models: ["expression-1"]`), the payload built by this
plugin's own `_send_json_request`:
1.8.2 model="essence" sends no model, mode=cpu
-> HTTP 200, session STARTED and served as expression-1
A session started as a model the caller did not ask for, and nothing said so.
this model="expression-2" sends model="expression-2", mode=gpu
-> HTTP 409, "agent ...'s expression-2 model hasn't been generated
yet - add it with POST /v1/agent/{code}/models ..., then retry"
...before a renderer launches or a credit is spent. On an avatar that IS
prepared, the same call is answered `{"model": "expression-2",
"avatar_session_started": true}` where 1.8.2 was answered "expression-1".
model="essence-2" reaches the Essence 2 launcher, which names the bundle it
wants for that identity - so the string routes end to end.
The two first-generation names are unchanged and keep resolving to the
first-generation engines: driven against the same API, "expression" and
"essence" produce byte-identical payloads to 1.8.2 apart from the added field,
and the same responses.
`mode` is still sent, unchanged, for servers that predate `model`; the server
prefers the explicit field. The gpu/cpu split and the custom-endpoint request
format now read one property, `_is_expression_model`, so the two sites cannot
disagree about which family a session is - dispatching one way and billing the
other is the failure that shape invites.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vTGmNLyJSwj28bfFk5gG2
Devin review, and it is right: `NotGivenOr[BitHumanModel] = "essence"` lets a
caller pass NOT_GIVEN explicitly, and code forwarding an optional parameter does
that routinely. NotGiven defines `__repr__` as "NOT_GIVEN" and no `__str__`, so
`str(NOT_GIVEN)` is the literal string "NOT_GIVEN" (measured against the real
class in livekit/agents/types.py, not assumed). It would have been sent as the
model name and refused by the server, on a call the caller meant as "the
default".
Fixed at the ASSIGNMENT rather than at the reported read site, because there are
two readers and the reported one is not the only one at risk:
self._model = model # __init__
str(self._model).startswith("expression") # _is_expression_model
"model": str(self._model) # the session request
With NOT_GIVEN the second yields False and the session is dispatched "cpu". That
happens to agree with the essence default, so it is benign today β purely by
luck, and that property's own docstring says the two readers "must give the same
answer in both, or a session is dispatched one way and billed the other".
Patching only the JSON field would have left that agreement resting on a
coincidence. Normalising once makes both correct by construction.
Verified across every input the signature admits β NOT_GIVEN, essence,
essence-2, expression, expression-2 β that the model sent is always one of the
four valid names and that the dispatch mode matches it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134RFpn6JSLqWYJ88rJ9ohB
The first commit taught the CLOUD request to name the model. That left the
SELF-HOSTED path β a custom `api_url`, which routes to multipart/form-data β
still sending only livekit_url, livekit_token, room_name, async_mode,
avatar_image/avatar_image_url and avatar_id. No model. So a self-hosted worker
could not tell `expression` from `expression-2`: two engines that render
differently and meter on different lines, chosen by a server-side default rather
than by what the caller asked for. That is the same defect the JSON change
fixes, in the transport the change did not touch.
Adding it is additive for every existing deployment, verified against the live
worker rather than assumed: /launch reads the form into a dict and picks the
keys it knows, so an unrecognised field is ignored. Measured on the running
expression-2 worker β a form with no `model`, one with `model=expression-2`, and
one with a deliberately garbage field all return the identical 400 for the
missing required fields. No 422, no parse failure.
With this the model reaches the renderer on every path the plugin offers:
local engine read from the model file itself; the runtime routes
expression-2 to Expression2StreamingRuntime and derives the
billing line via billing_type_for_engine (verified in the
shipped bithuman 2.11.6 wheel, not assumed from the signature β
`create()` has no `model` argument and its billing_type default
names essence, but that default is not what a v2 engine uses)
self-hosted FormData (expression family) / JSON (essence family) β both now
carry `model`
cloud API JSON β carries `model`
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134RFpn6JSLqWYJ88rJ9ohB
Caught reviewing my own backward compatibility before asking for approval, and
it would have broken existing users.
The parameter defaulted to the literal "essence", and the two commits before this
one sent `model` on EVERY session. So a caller who had never heard of this
parameter β the whole installed base β would suddenly have been sending
`model: "essence"`. Naming a model is a PIN, not a hint: the API answers 400 and
names what the avatar can be served as if it was not prepared for that model. An
avatar prepared only as essence-2 would have gone from working to refused, on an
upgrade that touched none of its code.
So `model` now defaults to NOT_GIVEN and is sent only when the caller names one:
unset nothing is sent; BitHuman resolves the avatar exactly as it
did before this parameter existed
named sent, on both the cloud JSON path and the self-hosted
form-data path
Verified as equivalence, not by inspection: for a caller who names no model the
request body is byte-identical to pre-PR in both branches (`{"mode": "cpu"}` /
`{"mode": "gpu"}` with avatar_image), and the sentinel is never stringified, so
Devin's "NOT_GIVEN" can no longer be produced by any path at all β the earlier
normalise-to-"essence" fix is removed because it was the thing creating the
regression.
`_is_expression_model` gains the same guard, so the mode field and the model
field still agree on every input.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134RFpn6JSLqWYJ88rJ9ohB
CI type-check, both 3.10 and 3.13:
"AvatarSession" has no attribute "_model"; maybe "_mode"? x5
The previous commit rewrote the block that assigns `self._model` in order to
replace the explanation above it. The replacement kept the comment and dropped
the line it was explaining, so `self._model` was never assigned at all β an
AttributeError on the first session, not merely a typing complaint.
Two things did not catch it, and both are worth naming. My local mypy run could
not resolve the base class in this environment, so the attribute error was
invisible among ten pre-existing import-resolution errors that are identical
before and after the change. And the behaviour matrix I ran to "prove"
equivalence modelled the LOGIC in a separate script rather than exercising the
module β so it happily proved a property of code that was no longer there. CI,
which installs the package properly, found it in one run.
Added a structural check while fixing it: every `self._X` that is read must also
be assigned somewhere in the class. It reports `_model` missing on the broken
version and clean on this one (`_room` is inherited and shows in the green
control too, so it is not a new finding).
Behaviour is unchanged from the intent of the previous commit: `model` defaults
to NOT_GIVEN and is sent only when the caller names one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134RFpn6JSLqWYJ88rJ9ohB
|
Summary for a reviewer, since this picked up a few commits. What it does. BitHuman has two model families in two generations β Blast radius. Two files, both inside Backward compatibility β the part worth checking. That is deliberate and it is a correction I made to my own earlier commits in this PR. They defaulted the parameter to Review feedback. The one Devin finding (an explicit Verification.
CI. Happy to squash the history if you'd prefer a single commit. Thanks for taking a look. |
Three pages told a developer less than the product does. Each was written when it was true and never re-read against the thing it describes. 1. /sdk, Essence 2 / Android β "the step-by-step is not on the Android page yet", and it pointed at the CLI, Python or the cloud API instead. Measured against sdk/android.md on origin/main: that page carries `## Install ### Essence 2` (L91) with the dependency line, `## Minimal code ### Essence 2` (L175) with the imports and the render loop, `## Get a model`, `## Authentication`, and it links a whole worked app. The hub card was the only thing standing between a reader and all of it. 2. /sdk, Essence 2 / Web and /sdk/web β both described the downloadable benchmark package (one built-in identity, cannot render your own agent) as if that were the whole of Essence 2 in a browser, and /sdk/web listed `?render=local` for Expression 2 only. It is not model-specific: apps/agent-ui/src/lib/local-render/family.ts dispatches essence1, essence2 and expression2 to in-browser renderers and refuses only expression1, and sdk/performance.md's own Web / Essence 2 cell is measured on `<CODE>?render=local`. Measured anonymously on 2026-09-22, the public asset mirror answers HTTP 200 for 12 of 12 live Essence 2 identities and 11 of them publish the faster renderer's files (a bogus code answers HTTP 400, so a 200 means something). `https://www.bithuman.ai/A21SKT4314?render=local` redirects to the viewer with a session token whose payload reads `"model":"essence-2","anonymousEnabled":true`; a bogus code redirects to /not-found and answers 404. Both pages now say `?render=local` works for all three models that have an in-browser renderer, and Troubleshooting gains the two outcomes a reader actually meets: the page rewriting the URL to `?render=cloud` when an identity has no in-browser build, and the refusal message the renderer prints. No frame rate is repeated here β the performance page stays the one writer. 3. /sdk/livekit β three claims re-measured today: * "open upstream as livekit/agents#7366 and is not released yet" β #7366 MERGED 2026-09-22T02:23:29Z. Still unreleased (PyPI's newest livekit-plugins-bithuman is 1.8.2, 2026-09-15T18:16:50), so the page now says merged-and-unreleased rather than open. #7280 likewise merged 2026-09-15T22:08:47Z, hours after 1.8.2 was cut. * the Swift package pinned `.upToNextMajor(from: "2.14.0")` while livekit/client-sdk-swift's newest release is 2.17.0 (2026-09-14) β three minors stale, and graded by nothing here: check-versions-current's V6 tapPins only matches bithuman-product/homebrew-bithuman URLs. * "the local-essence and cloud-essence examples ship with this applied" β cloud-essence ships no tuned_publish.py and its agent.py names no publish option at all. It does not need one: it passes an `avatar_id`, so a bitHuman worker publishes the track. Only local-essence imports it. And the install line pins nothing, so the page now says which release it describes and how to check what you got. Everything the page claims about the install was re-run in throwaway venvs today: `pip install livekit-plugins-bithuman pillow` on 3.12 installs 1.8.2 + bithuman 2.11.6 + pillow 12.3.0; without pillow the first import dies on `ModuleNotFoundError: No module named 'PIL'`; on 3.10 and on 3.14 the marker is false, no wheel installs and the import dies on `No module named 'cv2'`. `AvatarSession.__init__` still declares `model: Literal['expression','essence'] = 'essence'`, and the documented monkey-patch still matches AvatarRunner._publish_track line for line, with every private attribute it touches set in __init__. "There is no npm package" also still holds: the registry's search for `bithuman` returns 0 results and six candidate names answer 404, with livekit-client returning 200 as the control. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0134RFpn6JSLqWYJ88rJ9ohB
AvatarSession(model=...)accepts only"expression"and"essence", and the request it sends names no model at all β justmode: gpu|cpu, which can distinguish exactly two engines. Which model a second-generation avatar is served as therefore depends on a server-side default rather than on what the caller asked for, and the caller has no way to say.Measured against BitHuman's live API
One account, and an avatar prepared for only one model (
supported_models: ["expression-1"]). The payload on the wire is the one this plugin's own_send_json_requestbuilds β only the transport wrapper was replaced so the response the plugin discards could be read.model="essence"mode=cpuexpression-1model="expression-2"model="expression-2",mode=gpuPOST /v1/agent/{code}/models, then retry"A session started as a model the caller did not ask for, and nothing said so. The new refusal arrives before a renderer launches or a credit is spent.
On an avatar that is prepared, the same call is answered
{"model": "expression-2", "avatar_session_started": true}where 1.8.2 was answered"expression-1".model="essence-2"reaches the Essence 2 launcher, which names the bundle it wants for that identity β so the string routes end to end.Backward compatibility
The two first-generation names are unchanged and keep resolving to the first-generation engines: driven against the same API,
"expression"and"essence"produce payloads identical to 1.8.2 apart from the added field, and the same responses.modeis still sent, unchanged, for servers that predatemodel; the server prefers the explicit field.The gpu/cpu split and the custom-endpoint request format now read one property,
_is_expression_model, so the two sites cannot disagree about which family a session is β dispatching one way and billing the other is the failure that shape invites.Patch bump, 1.8.2 β 1.8.3, per
AGENTS.md.π€ Generated with Claude Code
https://claude.ai/code/session_012vTGmNLyJSwj28bfFk5gG2
Coverage after review (2026-09-21)
Two follow-ups from the Devin review, and one gap it surfaced by making me re-read the whole path.
1. An explicit
NOT_GIVENno longer serializes as a model name.NotGiven.__repr__is"NOT_GIVEN"and the class defines no__str__, sostr(NOT_GIVEN)is that literal string. Normalised once at the assignment rather than at the reported read site, because there are two readers β_is_expression_modeland themodelfield β and that property's docstring requires them to agree. Patching only the JSON field would have left the agreement resting on a coincidence.2. The self-hosted path now names the model too. The first commit taught only the cloud JSON request. A custom
api_urlroutes tomultipart/form-data, which sent nomodelat all β so a self-hosted worker could not distinguishexpressionfromexpression-2, two engines that render differently and meter on different lines.Verified additive rather than assumed:
/launchreads the form into a dict and picks known keys, so an unrecognised field is ignored. Measured against a running expression-2 worker β a form with nomodel, one withmodel=expression-2, and one with a deliberately garbage field all return the identical400for the missing required fields. No422, no parse failure, so no existing deployment changes behaviour.Where the model reaches the renderer
model_path)multipart/form-datamodelmodelmodelLocal mode was checked against the shipped
bithuman2.11.6 wheel rather than inferred from the signature:AsyncBithuman.create()takes nomodelargument and itsbilling_typedefault names essence, but that default is not what a second-generation engine ends up using β the loader reads the engine from the model header and callsbilling_type_for_engine().