Skip to content

str.isidentifier() does not work with non-BMP non-canonicalized strings on Windows #84776

Closed
Closed
str.isidentifier() does not work with non-BMP non-canonicalized strings on Windows#84776
Labels
3.9 (EOL)end of lifeinterpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

Description

@serhiy-storchaka
serhiy-storchaka
opened on May 11, 2020
BPO 40596
Nosy @vstinner, @serhiy-storchaka
PRs
  • bpo-40596: Fix str.isidentifier() for non-canonicalized strings containing non-BMP characters on Windows. #20035
  • bpo-40596: Fix str.isidentifier() for non-canonicalized strings containing non-BMP characters on Windows. #20053
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-05-12.13:19:12.227>
created_at = <Date 2020-05-11.17:50:16.125>
labels = ['interpreter-core', 'type-bug', '3.9']
title = 'str.isidentifier() does not work with non-BMP non-canonicalized strings on Windows'
updated_at = <Date 2020-05-12.17:27:57.214>
user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2020-05-12.17:27:57.214>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-05-12.13:19:12.227>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2020-05-11.17:50:16.125>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 40596
keywords = ['patch']
message_count = 6.0
messages = ['368637', '368651', '368652', '368705', '368729', '368739']
nosy_count = 2.0
nosy_names = ['vstinner', 'serhiy.storchaka']
pr_nums = ['20035', '20053']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40596'
versions = ['Python 3.9']

    Activity

    1. serhiy-storchaka commented on May 11, 2020

      @serhiy-storchaka
      serhiy-storchaka
      on May 11, 2020
      MemberAuthor
      >>> import _testcapi
>>> u = '\U0001d580\U0001d593\U0001d58e\U0001d588\U0001d594\U0001d589\U0001d58a'
>>> u.isidentifier()
True
>>> _testcapi.unicode_legacy_string(u).isidentifier()
False
    2. serhiy-storchaka
      added
      3.9 (EOL)end of life
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      type-bugAn unexpected behavior, bug, or error
      on May 11, 2020

    3. vstinner commented on May 11, 2020

      @vstinner
      vstinner
      on May 11, 2020
      Member

      It's maybe time to speed up the deprecation of the legacy C API using Py_UNICODE...

    4. vstinner commented on May 11, 2020

      @vstinner
      vstinner
      on May 11, 2020
      Member

      My previous change on this function:

      commit f3e7ea5
      Author: Victor Stinner <vstinner@python.org>
      Date: Tue Feb 11 14:29:33 2020 +0100

      bpo-39500: Document PyUnicode_IsIdentifier() function (GH-18397)

PyUnicode_IsIdentifier() does not call Py_FatalError() anymore if the
string is not ready.

    5. serhiy-storchaka commented on May 12, 2020

      @serhiy-storchaka
      serhiy-storchaka
      on May 12, 2020
      MemberAuthor

      I am not sure that changes in bpo-39500 was correct. It is easier to catch a bug if crash consistently when you pass a non-canonicalized strings then if silently return a wrong result for specific input on particular platform.

      Alternatively, you could reimplement correct handling of surrogate pairs in PyUnicode_IsIdentifier().

    6. serhiy-storchaka commented on May 12, 2020

      @serhiy-storchaka
      serhiy-storchaka
      on May 12, 2020
      MemberAuthor

      New changeset 5650e76 by Serhiy Storchaka in branch 'master':
      bpo-40596: Fix str.isidentifier() for non-canonicalized strings containing non-BMP characters on Windows. (GH-20053)
      5650e76

    7. serhiy-storchaka
      closed this as completedon May 12, 2020
    8. serhiy-storchaka
      closed this as completedon May 12, 2020

    9. vstinner commented on May 12, 2020

      @vstinner
      vstinner
      on May 12, 2020
      Member

      Thanks for the fix Serhiy!

    10. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.9 (EOL)end of lifeinterpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions