Add spec-compliance test suite, close WHATWG/IANA alias gaps - #366

Summary

2a54c68 fixed a bug where chardet returned utf-16-le/utf-16-be for BOM-prefixed input. Python's -le/-be codecs don't strip the BOM on decode, so users got a leading U+FEFF in their decoded strings. We caught that by inspection, not by a test, which felt like a near-miss worth addressing.

This PR adds a 5-file spec-compliance test suite, and closes the interop gaps it surfaced in the registry along the way.

What the suite tests

Four dimensions of output correctness, all one-way (spec -> chardet):

• Python decode round-trip. Every name detect() can return must decode cleanly via bytes.decode() without leaking U+FEFF or raising LookupError. 516 parametrized cases across all 86 registry encodings, covering the top-pick name, the compat_names=True drop-in contract, all aliases via lookup_encoding, and every detect_all() candidate.
• WHATWG Encoding Standard. Every <meta charset=\"X\"> label WHATWG specifies must route through chardet's markup stage to an expected canonical. Tests are end-to-end via the real chardet.detect() pipeline, not just lookup_encoding in isolation, so we're exercising what real users hit when they parse HTML.
• IANA Character Sets registry. Every IANA-preferred MIME name chardet aims to support must resolve to the expected canonical.
• Unicode/RFC conformance. pipeline.bom._BOMS matches WHATWG's decode-algorithm sniff set exactly; pipeline.utf8.detect_utf8 matches RFC 3629 boundary cases (overlong rejections, surrogate rejections, U+10FFFF); UTF-16 surrogate pairs round-trip per RFC 2781.

Plus a dedicated tests/test_python_stdlib_limitations.py that pins CPython codec behaviors chardet depends on (utf-16 BOM stripping, utf-8-sig, and the gb18030 leak, see below). These aren't tests of chardet's own code: they're forward-looking regression guards for upstream invariants.

Spec data is hardcoded as Python constants with # Source: URL, Snapshot: YYYY-MM-DD comments. WHATWG's encodings.json hasn't changed since 2020-05-06 and IANA's registry was last updated 2024-06-06, so a fetcher script with freshness automation felt like overkill for data that changes once per decade.

Registry alias additions

The WHATWG and IANA tests surfaced real interop gaps: 57 of 58 failing WHATWG labels weren't recognized by Python's stdlib codec table either, which meant any <meta charset=\"x-cp1252\"> style declaration was being silently dropped by chardet's markup stage. Rather than allow-list the gaps, I added the missing aliases directly:

• WHATWG web labels (x-cp1250..x-cp1258, x-sjis, x-gbk, x-x-big5, compact no-hyphen iso88591..iso885916, dos-874, visual/logical Hebrew bidi variants, etc.)
• IANA csXxx identifiers (csUTF8, csISO885913..csISO885916, cswindows1250..cswindows1258, csGB18030, csBig5HKSCS, etc.)
• IBM 5-digit forms (cp01140, ibm01140, csibm01140 on the existing cp1140 entry)
• Archaic Unicode 1.1 UTF-8 labels (unicode-1-1-utf-8, unicode11utf8, unicode20utf8, x-unicode20utf8). Rarely used in practice but WHATWG-registered.

Every alias was driven by a failing test, not speculative. Two labels I initially added and then dropped during review: \"chinese\" and \"korean\". Both are semantically lossy (chinese would route Traditional Chinese content at gb18030, which silently mis-decodes Big5 bytes to garbled output), and Python's codec fallback resolves them both anyway via gb2312 and euc_kr, so chardet's own table doesn't need to bless the ambiguity.

One new test in tests/test_registry.py (test_aliases_are_unique_across_entries) catches the kind of cross-entry collision bug that a careless refactor could introduce, plus two referential-integrity tests that make sure _WHATWG_TO_CHARDET and IANA_ENTRIES never end up pointing at a canonical that's been renamed out from under them.

What this PR deliberately does NOT do

No GB18030 BOM detection. The byte sequence 0x84 0x31 0x95 0x33 is GB18030's encoding of U+FEFF, and Python's gb18030 codec doesn't strip it on decode. That's the same bug class as the UTF-16 issue, but after some research:

• WHATWG's decode algorithm doesn't sniff the GB18030 BOM either, only UTF-8, UTF-16BE, and UTF-16LE
• CPython ships no gb18030-sig variant, so there's no alternative codec name we could return to get BOM stripping
• Treating this byte sequence as a signature is non-standard (no spec body does)

Instead, tests/test_python_stdlib_limitations.py::test_python_gb18030_bom_leak_still_present pins Python's current leaking behavior. If CPython ever starts stripping the BOM, that test fails, and we reconsider whether chardet should add detection. Users who encounter real-world BOM-prefixed GB18030 content need to lstrip('\\ufeff') themselves after decoding.

Test plan

• uv run python -m pytest -n auto -> 9271 passed, 28 xfailed, 0 failed (the 28 xfailed are pre-existing known-failure corpus entries in test_accuracy.py)
• uv run python -m pytest tests/test_accuracy.py -n auto -> 7523 passed, 28 xfailed, 0 failed. Zero regressions from the ~90 alias additions.
• uv run ruff check . + uv run ruff format --check . -> clean
• Time benchmark (scripts/benchmark_time.py --encoding-era all) on the 2517-file corpus: baseline 4980ms -> post-change 5003ms (+0.46%, well within run-to-run noise). lookup_encoding is @functools.cache-decorated, so the expanded alias count only matters for first-call cost on previously-unseen names, and that's at most 24us in the worst case.

Fixes the class of bug that slipped through in 2a54c68, not any specific new user report.

🤖 Generated with Claude Code