fix(file): Let legacy-encoded text reach the encoding detector - #1879

What

Fixes #1878

A legacy-encoded source file never reaches the encoding detector that exists for it, because the binary content check in front of that detector always votes "binary" for exactly those files. isbinaryfile looks at the first 512 bytes and counts every byte above 0x7F that is not part of a well-formed UTF-8 sequence as suspicious, bailing above a 10% ratio. Double-byte text is almost entirely such bytes, so Shift-JIS / EUC-KR / GBK sources trip that ratio every time and return before the jschardet + iconv-lite slow path runs.

Measured on a fixture where the same Japanese text exists once as UTF-8 and once as Shift-JIS (Total Files for 5 files on disk):

before (9f01703a): 3 files detected as binary by content inspection:
                   1. notes/euckr.txt  2. notes/gbk.txt  3. src/Big.java
                   Total Files: 2 files | 424 tokens

after:             Total Files: 5 files | 1,656 tokens

src/Utf8.java packed in both runs; its Shift-JIS twin src/Big.java only packs after the fix — decoded, no replacement characters (\uFFFD count is 0 in both outputs).

How

When the content check returns binary, the detector gets a second opinion before the file is dropped, and the verdict is overturned only on an unambiguous win:

• detection must name an encoding iconv actually supports and report confidence ≥ 0.9,
• the decode must need no replacement characters and contain no C0 control (beyond tab/LF/CR) or DEL — real text in these encodings never does, because their non-ASCII bytes stay at 0x80 and above,
• the result must look like language rather than a replay: structural whitespace, or at least 4 distinct characters.

The thresholds come from measurement, not from the number that made the fixture pass. On the same files as above: SHIFT_JIS 1.00, EUC-KR 0.99, GB2312 0.99, windows-1252 0.95. Ambiguous input stays exactly as it is today.

The third condition earned its place. A buffer with no control bytes clears isbinaryfile and then clears the confidence floor too — a replayed 2-byte cycle such as 0xB0 0xA1 decodes as EUC-KR at 0.99 — so without it, high-byte filler reaches the pack. Measured over 289 inputs that all get past the extension check because they are named .data:

The three layers do distinct work, which is why all three stay: the real binaries are stopped by the control-character and replacement-character checks (with the confidence floor switched off entirely they still score 0/120; the highest confidence observed on one was 0.88), and the filler is stopped by the diversity check, since it scores 0.99 and decodes cleanly. Every cycle that cleared 0.9 decoded to at most 2 distinct characters, while the shortest realistic separator-free CJK sample needs 7 — hence a floor of 4, and hence the "or": requiring whitespace alone would have dropped Chinese and Korean prose that has no ASCII in it at all.

Compatibility

Strictly additive on the packed set. The new branch is inside if (await isBinaryFile(buffer)), which previously returned immediately, so no file that already got packed is re-decoded or re-encoded here — the only outputs that change are ones that were missing a file.

Worth tying off the prior art: #752 (closed) reached this same branch — the maintainer called it "a false positive in our binary file detection" and shipped the visibility half in v1.4.0, which is the 3 files detected as binary by content inspection report this repro prints. The false positive itself was never corrected, so undecodable-by-ratio text still gets skipped; this PR is the other half of that issue, and it is why #1878's fixture reports files that are text.

The skip-reason taxonomy is unchanged (binary-extension / binary-content / size-limit / encoding-error); a rescued file simply stops appearing in the report.

Tests

Seven new cases in tests/core/file/fileRead.test.ts: Shift-JIS, EUC-KR and GBK text that the content check rejects must still be read; separator-free CJK prose must still be read; and three shapes must stay binary-content (high bytes with control characters, a replayed byte pair, a 0xFF padded block) so the rescue cannot become a general "decode anything with high bytes" rule.

• npm run test: 1827 passed | 20 skipped, 0 failed (152 files)
• npm run lint: clean — biome reports "No fixes applied" on the changed files, tsc -p tsconfig.build.json --noEmit exits 0

Checklist

• Run npm run test
• Run npm run lint