Skip to content

fix(serve): a lone surrogate in the body was a 500 instead of a 400 - #454

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/serve-surrogate
Sep 29, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/serve-surrogate

Conversation

@PerryLink

Copy link
Copy Markdown
Contributor

fix(serve): a lone surrogate in the body was a 500 instead of a 400

A \udXXX escape with no pair is legal JSON that json.loads accepts, and it builds a str
holding a code point that cannot be encoded as UTF-8. The tokenizer then raised from inside
build_sequence, and serve maps only ValueError to 422, so a malformed string in the
client's own body came back as a server fault:

{"state":"\ud800", ...}                       -> HTTP 500 {"detail":"inference failed"}
{"state":"ok", "instructions":"\udfff", ...}  -> HTTP 500 {"detail":"inference failed"}
{"criteria":["\ud800","b"], ...}              -> HTTP 500 {"detail":"inference failed"}

# the same request with a real emoji, which is a *paired* surrogate
{"state":"hi 😀", ...}                        -> HTTP 200

The 500 also wrote a traceback into the operator's log for every such request — TypeError: TextEncodeInput must be Union[TextInputSequence, ...] from the tokenizer — so a caller could
fill the log with server-looking faults by sending one escape. This is the same class as the
nested-choice-label fix: a caller error reaching an HTTP client as a server fault because the
exception type is not one serve classifies.

A body that survives json.loads with an unpaired surrogate is rejected at parse time, beside
the other body-shape checks, with a message naming the cause:

{"state":"\ud800", ...}  -> HTTP 400
  {"detail":"request body contains an unpaired surrogate escape; those cannot be encoded as UTF-8"}

Deliberate choices tested:

  • Paired surrogates are left alone. json.loads combines a \ud83d\ude00 pair into one astral character before anything here sees it, so an emoji never reaches the check and still returns 200 — asserted in the test, because a fix that rejected surrogates by code point rather than by unpairedness would break every emoji in a state.
  • The check walks the parsed body, not the raw bytes. Testing the raw bytes for \ud would reject a request that merely mentions the escape in text; walking the parsed value tests what the tokenizer will actually receive. Dict keys are walked as well as values, since a criteria label can be a key.
  • 400, at parse time, next to the other shape checks. The neighbouring checks in the same handler use 400 for a malformed body, and it belongs before _check_request_limits and _resolve_model so nothing downstream sees the value.
  • No encode("utf-8", "replace") sanitising. Replacing the character would answer a question about text the client did not send, which is the failure fix(serve): a request with no state was answered about the literal text null #427 just fixed one case of; a caller that sent a broken string should be told so.
# with the guard removed
FAILED tests/test_serve.py::test_an_unpaired_surrogate_is_a_caller_error_not_a_server_fault
1 failed, 26 passed

# with it
$ python -m pytest tests/test_serve.py -q        27 passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/

Known limitation: only /v1/systemone is checked. The OpenAI-compatible routes in the open
PR #281 parse their own bodies, so whether the same escape reaches the tokenizer there is that
PR's question and I have not verified it.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for finding this one, a lone surrogate turning a caller mistake into a 500 is a real bug and the emoji case in the test is well thought out. Two changes before merge. The recursive walk raises RecursionError on nested states that json.loads accepts (depth 600 now returns 500, while main returns 200), so please make it iterative with an explicit stack, or catch RecursionError as a 400 the way #401 does. Please also run it after the size checks, or use a C-speed test such as a precompiled regex over each string, since a near-cap body currently spends about 170 ms on the event loop.

A `\udXXX` escape with no pair is legal JSON that `json.loads` accepts, and it builds a `str`
holding a code point that cannot be encoded as UTF-8. The tokenizer then raised from inside
`build_sequence`, and `serve` maps only `ValueError` to 422, so a malformed string in the
client's own body came back as a server fault:

```
{"state":"\ud800", ...}                       -> HTTP 500 {"detail":"inference failed"}
{"state":"ok", "instructions":"\udfff", ...}  -> HTTP 500 {"detail":"inference failed"}
{"criteria":["\ud800","b"], ...}              -> HTTP 500 {"detail":"inference failed"}

# the same request with a real emoji, which is a *paired* surrogate
{"state":"hi 😀", ...}                        -> HTTP 200
```

The 500 also wrote a traceback into the operator's log for every such request — `TypeError:
TextEncodeInput must be Union[TextInputSequence, ...]` from the tokenizer — so a caller could
fill the log with server-looking faults by sending one escape.

The body is now rejected at parse time with a message naming the cause:

```
{"state":"\ud800", ...}  -> HTTP 400
  {"detail":"request body contains an unpaired surrogate escape; those cannot be encoded as UTF-8"}
```

## The two problems the review found in the first version

**It recursed.** `json.loads` accepts nesting far deeper than Python's default recursion limit,
so a recursive walk turned a body the parser handles into a `RecursionError` — one 500 replaced
by another:

```
depth=200   -> HTTP 200
depth=600   -> HTTP 500   (before)   -> HTTP 200 (after)
depth=1200  -> HTTP 500   (before)   -> HTTP 200 (after)
```

The walk now uses an explicit stack, so the depth the parser accepts is the depth this handles,
and a test pins 200/600/1200.

**It was slow.** The per-character Python loop cost ~170 ms of event-loop time on a near-cap
body. It is now one precompiled regex scanned at C speed:

```
per-char loop    1.6 ms per 49k-char string
compiled regex   0.1 ms per 49k-char string     (~16x)
```

and it runs **after** `_check_request_limits`, so an oversized body is refused before anything
walks it and `MAX_STATE_CHARS`/`MAX_QUESTIONS` bound what the walk can reach.

Deliberate choices tested:

- **Paired surrogates are left alone.** `json.loads` combines a `\ud83d\ude00` pair into one astral character before anything here sees it, so an emoji never reaches the check and still returns 200 — asserted, because a fix that rejected surrogates by code point rather than by unpairedness would break every emoji in a state.
- **The check walks the parsed body, not the raw bytes.** Testing the raw bytes for `\ud` would reject a request that merely *mentions* the escape in text; walking the parsed value tests what the tokenizer will actually receive. Dict keys are walked as well as values, since a criteria label can be a key.
- **400, at parse time, next to the other shape checks.** The neighbouring checks in the same handler use 400 for a malformed body.
- **No `encode("utf-8", "replace")` sanitising.** Replacing the character would answer a question about text the client did not send; a caller that sent a broken string should be told so.

```
$ python -m pytest tests/test_serve.py -q        32 passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

Known limitation: only `/v1/systemone` is checked. The OpenAI-compatible routes in PR NandhaKishorM#281 parse
their own bodies, so whether the same escape reaches the tokenizer there is that PR's question
and I have not verified it.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Passed checks and merged. Thank you @PerryLink! It ships in 0.3.22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants