Skip to content

Bugs and inconsistencies in unicodedata #80667

Closed
Closed
Bugs and inconsistencies in unicodedata#80667
Assignees
serhiy-storchaka
Labels
3.13only security fixes3.14bugs and security fixes3.15bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytopic-unicodetype-bugAn unexpected behavior, bug, or error

Description

@dscorbett
dscorbett
mannequin
opened on Mar 30, 2019
BPO 36486
Nosy @malemburg, @vstinner, @ezio-melotti, @dscorbett

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-03-30.14:41:13.171>
labels = ['3.8', 'type-bug', '3.7', 'expert-unicode', 'docs']
title = 'Bugs and inconsistencies in unicodedata'
updated_at = <Date 2019-04-05.18:28:55.383>
user = 'https://github.com/dscorbett'

bugs.python.org fields:

activity = <Date 2019-04-05.18:28:55.383>
actor = 'terry.reedy'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Unicode']
creation = <Date 2019-03-30.14:41:13.171>
creator = 'dscorbett'
dependencies = []
files = []
hgrepos = []
issue_num = 36486
keywords = []
message_count = 1.0
messages = ['339203']
nosy_count = 5.0
nosy_names = ['lemburg', 'vstinner', 'ezio.melotti', 'docs@python', 'dscorbett']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36486'
versions = ['Python 3.7', 'Python 3.8']

Linked PRs

Activity

  1. dscorbett commented on Mar 30, 2019

    @dscorbett
    dscorbettmannequin
    on Mar 30, 2019
    MannequinAuthor

    In unicodedata, the functions lookup and name have some bugs and inconsistencies.

    lookup matches case-insensitively, except for the algorithmic names of Hangul syllables and CJK unified ideographs, which must be in all caps. The documentation does not explain how character names are fuzzily matched.

    lookup accepts names like “CJK UNIFIED IDEOGRAPH-04E00”, where the code point has a leading zero.

    lookup and name don’t implement rule NR2, defined in chapter 4 of Unicode, for Tangut ideographs’ names.

  2. dscorbett
    added
    3.7 (EOL)end of life
    on Mar 30, 2019
  3. dscorbett
    assigned
    docspython
    on Mar 30, 2019
  4. dscorbett
    added
    docsDocumentation in the Doc dir
    topic-unicode
    type-bugAn unexpected behavior, bug, or error
    on Mar 30, 2019
  5. terryjreedy
    added
    3.8 (EOL)end of life
    on Apr 5, 2019
  6. ezio-melotti
    transferred this issue fromon Apr 10, 2022

  7. cfbolz commented on Oct 10, 2022

    @cfbolz
    cfbolz
    on Oct 10, 2022
    Contributor

    just hit the difference between CJK UNIFIED IDEOGAPH-*, HANGUL SYLLABLE * and other unicode names again:

    >>> "\N{SNOWMAN}\N{SNOWmAn}\N{sNoWMAN}\N{snowman}"
'☃☃☃☃'
>>> # but:
>>> "\N{CJK UNIFIED IDEOGRAPH-3400}"
'㐀'
>>> "\N{cjk UNIFIED IDEOGRAPH-3400}"
  File "<stdin>", line 1
    "\N{cjk UNIFIED IDEOGRAPH-3400}"
                                    ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 0-29: unknown Unicode character name
  8. StanFromIreland
    added
    stdlibStandard Library Python modules in the Lib/ directory
    and removed
    docsDocumentation in the Doc dir
    3.8 (EOL)end of life
    3.7 (EOL)end of life
    on Sep 26, 2025
  9. serhiy-storchaka
    added a commit that references this issue on Feb 12, 2026

    gh-80667: Fix case-sensitivity of some Unicode literal escapes (GH-10…

    e66f4a5
  10. miss-islington
    added a commit that references this issue on Feb 12, 2026

    pythongh-80667: Fix case-sensitivity of some Unicode literal escapes (p…

    01b9907

  11. 6 remaining items

  12. serhiy-storchaka
    added 3 commits that reference this issue on Feb 16, 2026

    gh-80667: Fix Tangut ideographs names in unicodedata (GH-144789)

    8b7b5a9

    [3.14] pythongh-80667: Fix Tangut ideographs names in unicodedata (py…

    11858e8

    [3.14] gh-80667: Fix lookup for Tangut ideographs in unicodedata (GH-…

    bcabbd0
  13. miss-islington
    added a commit that references this issue on Feb 16, 2026

    [3.14] pythongh-80667: Fix lookup for Tangut ideographs in unicodedata (

    7994a9e
  14. serhiy-storchaka
    added a commit that references this issue on Feb 16, 2026

    [3.13] gh-80667: Fix lookup for Tangut ideographs in unicodedata (GH-…

    ba0bab4

  15. serhiy-storchaka commented on Feb 16, 2026

    @serhiy-storchaka
    serhiy-storchaka
    on Feb 16, 2026
    Member

    Thank you for your report, @dscorbett, and for your PRs, @wismill and @SnoopJ. My apologies that it took so long to review them -- we have few Unicode experts -- but finally this was resolved.

    3.13 and 3.14 will only accept the Tangut ideographs' names, but not produce them. 3.15 will also produce them.

    Egyptian hieroglyphs, Khitan small script characters and Nushu characters were already supported, because their names were listed explicitly in UnicodeData.txt. But this gives us opprtunity for optimization, see #144882.

  16. serhiy-storchaka
    closed this as completedon Feb 16, 2026
  17. serhiy-storchaka
    added
    3.13only security fixes
    3.14bugs and security fixes
    3.15bugs and security fixes
    on Feb 16, 2026

  18. SnoopJ commented on Feb 16, 2026

    @SnoopJ
    SnoopJ
    on Feb 16, 2026
    Contributor

    Thanks @serhiy-storchaka, glad to be of some assistance! I'm glad to see your follow-up issue as well, I was wondering about further improvements when dealing with named ranges like that as I looked over your draft PR.

  19. brijkapadia
    added 2 commits that reference this issue on Feb 28, 2026

    pythongh-80667: Fix case-sensitivity of some Unicode literal escapes (p…

    669df2e

    pythongh-80667: Fix Tangut ideographs names in unicodedata (pythonGH-…

    74ad485
  20. ljfp
    added 2 commits that reference this issue on Apr 25, 2026

    pythongh-80667: Fix case-sensitivity of some Unicode literal escapes (p…

    0803946

    pythongh-80667: Fix Tangut ideographs names in unicodedata (pythonGH-…

    2e95075
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.13only security fixes3.14bugs and security fixes3.15bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytopic-unicodetype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions