Make tokenize recognize Other_ID_Start and Other_ID_Continue chars

@terryjreedy

BPO	24194
Nosy	@terryjreedy, @meadori, @iritkatriel
Dependencies	bpo-12731: python lib re uses obsolete sense of \w in full violation of UTS#18 RL1.2a bpo-12734: Request for property support in Python re lib
Files	issue24194-v0.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2015-05-14.13:00:27.728>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'Make tokenize recognize Other_ID_Start and Other_ID_Continue chars'
updated_at = <Date 2022-01-17.11:25:14.419>
user = 'https://bugs.python.org/JoshuaLandau'

bugs.python.org fields:

activity = <Date 2022-01-17.11:25:14.419>
actor = 'vstinner'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2015-05-14.13:00:27.728>
creator = 'Joshua.Landau'
dependencies = ['12731', '12734']
files = ['42808']
hgrepos = []
issue_num = 24194
keywords = ['patch']
message_count = 5.0
messages = ['243188', '265286', '313851', '410718', '410731']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'meador.inge', 'Joshua.Landau', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24194'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

This is valid:

    ℘· = 1
    print(℘·)
    #>>> 1

But this gives an error token:

    from io import BytesIO
    from tokenize import tokenize

    stream = BytesIO("℘·".encode("utf-8"))
    print(*tokenize(stream.read), sep="\n")
    #>>> TokenInfo(type=56 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='')
    #>>> TokenInfo(type=53 (ERRORTOKEN), string='℘', start=(1, 0), end=(1, 1), line='℘·')
    #>>> TokenInfo(type=53 (ERRORTOKEN), string='·', start=(1, 1), end=(1, 2), line='℘·')
    #>>> TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')

This is a continuation of http://bugs.python.org/issue9712. I'm not able to reopen the issue, so I thought I should report it anew.

It is tokenize that is wrong - Other_ID_Start and Other_ID_Continue are documented to be valid:

https://docs.python.org/3.5/reference/lexical_analysis.html#identifiers

Attached is a first cut patch for this. (CC'd haypo as a unicode expert).

I closed bpo-1693050 as a duplicate of bpo-12731 (the /w issue). I left bpo-9712 closed and closed bpo-32987 and marked both as duplicates of this.

In msg313814 of the latter, Serhiy indicates which start and continue identifier characters are currently matched by \W for re and regex. He gives there a fix for this that he says requires the /w issue to be fixed. It is similar to the posted patch. He says that without \w fixed, another 2000+ chars need to be added. Perhaps the v0 patch needs more tests (I don't know.)

He also says that re support for properties, bpo-12734, would make things even better.

Three of the characters in the patch are too obscure for Firefox on Window2 and print as boxes. Some others I do not recognize. And I could not type any of them. I thought we had a policy of using \u or \U escapes even in tests to avoid such problems. (I notice that there are already non-ascii chars in the context.)

Reproduced on 3.11.

Udated doc link, which appears to be same:
https://docs.python.org/3.11/reference/lexical_analysis.html#identifiers

Updated property list linked in above:
https://www.unicode.org/Public/14.0.0/ucd/PropList.txt

Relevant content for this issue:

1885..1886 ; Other_ID_Start # Mn [2] MONGOLIAN LETTER ALI GALI BALUDA..MONGOLIAN LETTER ALI GALI THREE BALUDA

2118 ; Other_ID_Start # Sm SCRIPT CAPITAL P
212E ; Other_ID_Start # So ESTIMATED SYMBOL
309B..309C ; Other_ID_Start # Sk [2] KATAKANA-HIRAGANA VOICED SOUND MARK..KATAKANA-HIRAGANA SEMI-VOICED SOUND MARK
# Total code points: 6

00B7 ; Other_ID_Continue # Po MIDDLE DOT
0387 ; Other_ID_Continue # Po GREEK ANO TELEIA
1369..1371 ; Other_ID_Continue # No [9] ETHIOPIC DIGIT ONE..ETHIOPIC DIGIT NINE
19DA ; Other_ID_Continue # No NEW TAI LUE THAM DIGIT ONE
# Total code points: 12

Codepoints of '℘·' opening example:
'0x2118' Other_Id_start Sm Script Capital P
'0xb7' Other_Id_continue P0 Middle dot

Except for the two Mongolian start characters, Meador's patch hardcodes the 'Other' characters, thereby adding them without waiting for re to be fixed. While this will miss new additions without manual updates, it is better than missing everything for however many years. I will make a PR with the additions and looks at the new tests.

I recently ran into this while working with asttokens, hence the PR above. IPython's tracebacks rely on asttokens, and entering this code:

In [1]: ℘℘ = 1
   ...: print(℘℘ .℘℘ )

crashes with a huge traceback which ends with:

Expected token NAME:None, got ERRORTOKEN:'℘' on line 2 col 11

Here's the workaround in that PR:

def patched_generate_tokens(original_tokens):
  # type: (Iterator[tokenize.TokenInfo]) -> Iterator[tokenize.TokenInfo]
  """
  Fixes tokens yielded by `tokenize.generate_tokens` to handle more non-ASCII characters in identifiers.
  Workaround for https://github.com/python/cpython/issues/68382.
  Should only be used when tokenizing a string that is known to be valid syntax,
  because it assumes that error tokens are not actually errors.
  Combines groups of consecutive NAME, NUMBER, and/or ERRORTOKEN tokens into a single NAME token.
  """
  group = []  # type: List[tokenize.TokenInfo]
  for tok in original_tokens:
    if (
        tok.type in (tokenize.NAME, tokenize.ERRORTOKEN, tokenize.NUMBER)
        # Only combine tokens if they have no whitespace in between
        and (not group or group[-1].end == tok.start)
    ):
      group.append(tok)
    else:
      for combined_token in combine_tokens(group):
        yield combined_token
      group = []
      yield tok
  for combined_token in combine_tokens(group):
    yield combined_token


def combine_tokens(group):
  # type: (List[tokenize.TokenInfo]) -> List[tokenize.TokenInfo]
  if not any(tok.type == tokenize.ERRORTOKEN for tok in group) or len({tok.line for tok in group}) != 1:
    return group
  return [
    tokenize.TokenInfo(
      type=tokenize.NAME,
      string="".join(t.string for t in group),
      start=group[0].start,
      end=group[-1].end,
      line=group[0].line,
    )
  ]

This seems to work but it's obviously a shaky hack. A proper fix would be great.

This has been fixed in 3.12.

Navigation Menu

Repository navigation

Make tokenize recognize Other_ID_Start and Other_ID_Continue chars #68382

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Make tokenize recognize Other_ID_Start and Other_ID_Continue chars #68382

Description

Activity

JoshuaLandau commented on May 14, 2015

meadori commented on May 11, 2016

terryjreedy commented on Mar 15, 2018

iritkatriel commented on Jan 16, 2022

terryjreedy commented on Jan 16, 2022

alexmojaki commented on Aug 1, 2022

lysnikolaou commented on Jun 19, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions