Read decimal digit values from the bundled character database - #8748

One of checkbox below must be checked.

• I did not use AI to write the code of this patch.
• This PR follows our AI policy

Summary

char_to_decimal in crates/common/src/str.rs carried its own sorted table of 590 decimal digit characters. It was added in #5643 because common had no way to reach the Unicode numeric data at the time — that data lived in stdlib, which sits above common in the dependency graph. The table's own comment said as much:

// TODO: this should probably live in a crate like unic or unicode-properties

#8211 extracted that data into rustpython-unicode, which sits below common, and made common depend on it. So the table can now go, in favour of Ucd::decimal — the same source unicodedata.decimal consults.

pub fn char_to_decimal(ch: char) -> Option<u8> {
    let value = rustpython_unicode::Ucd::new(true).decimal(CodePoint::from(ch))?;
    u8::try_from(value).ok()
}

This fixes a CPython incompatibility

The hand-written table was last updated for Unicode 10. The bundled UCD is Unicode 17, which has 180 more decimal digits across 18 scripts — Hanifi Rohingya, Wancho, Tangsa, Ol Onal, Kirat Rai, segmented digits and others. int() and float() rejected all of them, even though unicodedata.decimal in the same interpreter knew their value:

>>> int('𐴳')                      # U+10D33 HANIFI ROHINGYA DIGIT THREE
ValueError: invalid literal for int() with base 10: '𐴳'
>>> unicodedata.decimal('𐴳')
3

CPython accepts int('𐴳') and returns 3. After this patch RustPython does too, and the two lookups can no longer drift apart on a future Unicode update.

All 590 characters the old table covered keep their previous value (verified by diffing the table against the bundled UCD: 180 gained, 0 changed, 0 lost), so nothing that already worked changes.

Performance

About 1.5ns more per non-ASCII character, since it reads a property trie and a value table rather than binary-searching one 2.4KB array — int() over a 4000-digit non-ASCII string goes from 12.2 to 13.6 ns/char. ASCII input never reaches it: the sole caller, transform_decimal_and_space_to_ascii, returns early for ASCII strings, so ordinary numeric parsing is unaffected (5.6 → 5.4 ns/char, i.e. noise).

Tests

Adds char_to_decimal_values, covering ASCII, an Arabic-Indic digit the old table had, two digits it predated (U+10D35, U+1FBF5), and non-decimal characters (½, 가).

🤖 Generated with Claude Code