fix(lang): count letters no script range claims, instead of routing them to English - #169

Widen the Latin range in the shared script ranges to U+02B0 so the schwa 'ə' (U+0259, IPA Extensions) counts as a Latin letter. Uppercase 'Ə' (U+018F) already fell below the cutoff, so the same letter counted in one case and not the other. Since NandhaKishorM#169 the unclaimed 'ə' lands in "other", which reads Azerbaijani as partly non-Latin: "Müştəridən iki dəfə pul alınıb" reports non_latin_fraction 0.1111 and a lone 'ə' reports script "other" at 1.0, for text that is Latin script throughout.

Add an Azerbaijani function-word list and the schwa to the non-English diacritic set. The diacritic signal does not reach Azerbaijani typed without diacritics, which is common in support text, so that text still went to the English checkpoint. All 41 words are unique to the list, so it names Azerbaijani only on evidence no other list claims, per the rule added in NandhaKishorM#178.

Fold the dotted capital 'İ' to 'i' before lowercasing, after the NandhaKishorM#179 identifier mask. Python lowercases it to 'i' plus a combining dot above, which is not a word character, so 'İLƏ' split into 'i' and 'lə' and matched no word list; uppercase Azerbaijani now scores the same as lowercase. The remaining Azerbaijani letters were already handled correctly in both cases.

Add regression cases for the lone schwa, uppercase input with the dotted I, script profiles, text typed without diacritics, multilingual routing and an explicit English override. The routing suite reports 7 failures before the fix and 338 passes after it. A sweep over 23 states changes exactly one route, the de-diacriticised Azerbaijani case; English with links, French, Spanish, Brazilian Portuguese, Italian, German, Romanian, Polish, Turkish, Hindi, Japanese, Armenian, Cherokee, fullwidth Latin and digit-only input are unchanged.

Tests ran with a namespace loader and a stub torch that bypass the ML-dependent package initializer: test_router 338, test_email 46 and test_packaging 15 pass. test_criteria, test_decision_model, test_download and test_shortlist need transformers, safetensors or a real package import and were not run, and model inference was not exercised. ruff was not available locally. Byte compilation and git diff --check pass.

Widen the Latin range in the shared script ranges to U+02B0 so the schwa 'ə' (U+0259, IPA Extensions) counts as a Latin letter. Uppercase 'Ə' (U+018F) already fell below the cutoff, so the same letter counted in one case and not the other. Since #169 the unclaimed 'ə' lands in "other", which reads Azerbaijani as partly non-Latin: "Müştəridən iki dəfə pul alınıb" reports non_latin_fraction 0.1111 and a lone 'ə' reports script "other" at 1.0, for text that is Latin script throughout.

Add an Azerbaijani function-word list and the schwa to the non-English diacritic set. The diacritic signal does not reach Azerbaijani typed without diacritics, which is common in support text, so that text still went to the English checkpoint. All 41 words are unique to the list, so it names Azerbaijani only on evidence no other list claims, per the rule added in #178.

Fold the dotted capital 'İ' to 'i' before lowercasing, after the #179 identifier mask. Python lowercases it to 'i' plus a combining dot above, which is not a word character, so 'İLƏ' split into 'i' and 'lə' and matched no word list; uppercase Azerbaijani now scores the same as lowercase. The remaining Azerbaijani letters were already handled correctly in both cases.

Add regression cases for the lone schwa, uppercase input with the dotted I, script profiles, text typed without diacritics, multilingual routing and an explicit English override. The routing suite reports 7 failures before the fix and 338 passes after it. A sweep over 23 states changes exactly one route, the de-diacriticised Azerbaijani case; English with links, French, Spanish, Brazilian Portuguese, Italian, German, Romanian, Polish, Turkish, Hindi, Japanese, Armenian, Cherokee, fullwidth Latin and digit-only input are unchanged.

Tests ran with a namespace loader and a stub torch that bypass the ML-dependent package initializer: test_router 338, test_email 46 and test_packaging 15 pass. test_criteria, test_decision_model, test_download and test_shortlist need transformers, safetensors or a real package import and were not run, and model inference was not exercised. ruff was not available locally. Byte compilation and git diff --check pass.

Co-authored-by: Nandakishor <48623612+NandhaKishorM@users.noreply.github.com>

…hem to English (NandhaKishorM#169)

`detect_script` counts an alphabetic character only when one of `_SCRIPT_RANGES`
claims it. The loop had no `else`, so a character no range covers was counted
nowhere: text written only in an unlisted script produced a total of 0, was
reported as "unknown", and `analyse` treats "unknown" as English. It therefore
reached the English checkpoint -- which has no tokens for it -- with the reason
"no letters detected in state", for text that plainly has letters.

92,529 of Unicode's 136,104 alphabetic codepoints (68%) match no range, because
the table names only the scripts the checkpoints were measured on. That includes
the CJK ideograph extensions (60,331), the kana supplements, bopomofo, halfwidth
katakana, hangul jamo extended-A/B, and dozens of smaller scripts.

Confirmed on 573e5b6, all reporting unknown / is_english=True / model "english":
halfwidth katakana, bopomofo, kana supplement, CJK Ext-B, hangul jamo ext-A,
Cherokee, Mongolian, Syriac, Thaana, Tifinagh and Yi. Fullwidth Latin was also
read as an unlisted script rather than `latin`.

Both `_SCRIPT_RANGES` loops now count an unclaimed character under "other",
which is not "latin" and so routes to the multilingual checkpoint -- the safe
direction, matching the existing rule that an undecided Latin-script language is
not English. Fullwidth Latin (U+FF21-FF3A, U+FF41-FF5A) counts as `latin`.

`else` rather than a longer table because the defect is structural: no finite
list fixes it, and any list that tries is wrong again at the next Unicode release.

Letterless states keep the `unknown` path, so `route` still reports "no letters
detected" for empty, digits-only and emoji-only input.

tests/test_router.py: 23 new assertions. On 573e5b6 they give 166 passed,
23 failed; on this branch 189 passed, 0 failed. test_criteria.py 34/0,
test_email.py 11/0, test_shortlist.py 81/0 and test_decision_model.py unchanged.
ruff and compileall clean.

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>

…shorM#36)

Widen the Latin range in the shared script ranges to U+02B0 so the schwa 'ə' (U+0259, IPA Extensions) counts as a Latin letter. Uppercase 'Ə' (U+018F) already fell below the cutoff, so the same letter counted in one case and not the other. Since NandhaKishorM#169 the unclaimed 'ə' lands in "other", which reads Azerbaijani as partly non-Latin: "Müştəridən iki dəfə pul alınıb" reports non_latin_fraction 0.1111 and a lone 'ə' reports script "other" at 1.0, for text that is Latin script throughout.

Add an Azerbaijani function-word list and the schwa to the non-English diacritic set. The diacritic signal does not reach Azerbaijani typed without diacritics, which is common in support text, so that text still went to the English checkpoint. All 41 words are unique to the list, so it names Azerbaijani only on evidence no other list claims, per the rule added in NandhaKishorM#178.

Fold the dotted capital 'İ' to 'i' before lowercasing, after the NandhaKishorM#179 identifier mask. Python lowercases it to 'i' plus a combining dot above, which is not a word character, so 'İLƏ' split into 'i' and 'lə' and matched no word list; uppercase Azerbaijani now scores the same as lowercase. The remaining Azerbaijani letters were already handled correctly in both cases.

Add regression cases for the lone schwa, uppercase input with the dotted I, script profiles, text typed without diacritics, multilingual routing and an explicit English override. The routing suite reports 7 failures before the fix and 338 passes after it. A sweep over 23 states changes exactly one route, the de-diacriticised Azerbaijani case; English with links, French, Spanish, Brazilian Portuguese, Italian, German, Romanian, Polish, Turkish, Hindi, Japanese, Armenian, Cherokee, fullwidth Latin and digit-only input are unchanged.

Tests ran with a namespace loader and a stub torch that bypass the ML-dependent package initializer: test_router 338, test_email 46 and test_packaging 15 pass. test_criteria, test_decision_model, test_download and test_shortlist need transformers, safetensors or a real package import and were not run, and model inference was not exercised. ruff was not available locally. Byte compilation and git diff --check pass.

Co-authored-by: Nandakishor <48623612+NandhaKishorM@users.noreply.github.com>