Skip to content

Multiple lines f-string with non-ASCII breaks tokenize.generate_tokens in 3.12.4 #120343

Closed
Closed
Multiple lines f-string with non-ASCII breaks tokenize.generate_tokens in 3.12.4#120343
Assignees
pablogsallysnikolaou
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

Description

@leemars
leemars
opened on Jun 11, 2024

Bug report

Bug description:

import io
import tokenize

src = '''\
a = f"""
    Autorzy, którzy tą jednostkę mają wpisani jako AKTUALNA -- czyli"""
'''
tokens = list(tokenize.generate_tokens(io.StringIO(src).readline))

for token in tokens:
    print(token)

assert tokens[4].start == (2, 68), tokens[4].start

Python 3.12.3 (correct)

❯ python --version
Python 3.12.3

❯ python c.py
TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a = f"""\n')
TokenInfo(type=55 (OP), string='=', start=(1, 2), end=(1, 3), line='a = f"""\n')
TokenInfo(type=61 (FSTRING_START), string='f"""', start=(1, 4), end=(1, 8), line='a = f"""\n')
TokenInfo(type=62 (FSTRING_MIDDLE), string='\n    Autorzy, którzy tą jednostkę mają wpisani jako AKTUALNA -- czyli', start=(1, 8), end=(2, 68), line='a = f"""\n    Autorzy, którzy tą jednostkę mają wpisani jako AKTUALNA -- czyli"""\n')
TokenInfo(type=63 (FSTRING_END), string='"""', start=(2, 68), end=(2, 71), line='    Autorzy, którzy tą jednostkę mają wpisani jako AKTUALNA -- czyli"""\n')
TokenInfo(type=4 (NEWLINE), string='\n', start=(2, 71), end=(2, 72), line='    Autorzy, którzy tą jednostkę mają wpisani jako AKTUALNA -- czyli"""\n')
TokenInfo(type=0 (ENDMARKER), string='', start=(3, 0), end=(3, 0), line='')

Python 3.12.4 (broken)

❯ python --version
Python 3.12.4

❯ python c.py
TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a = f"""\n')
TokenInfo(type=55 (OP), string='=', start=(1, 2), end=(1, 3), line='a = f"""\n')
TokenInfo(type=61 (FSTRING_START), string='f"""', start=(1, 4), end=(1, 8), line='a = f"""\n')
TokenInfo(type=62 (FSTRING_MIDDLE), string='\n    Autorzy, którzy tą jednostkę mają wpisani jako AKTUALNA -- czyli', start=(1, 8), end=(2, 68), line='a = f"""\n    Autorzy, którzy tą jednostkę mają wpisani jako AKTUALNA -- czyli"""\n')
TokenInfo(type=63 (FSTRING_END), string='"""', start=(2, 72), end=(2, 75), line='    Autorzy, którzy tą jednostkę mają wpisani jako AKTUALNA -- czyli"""\n')
TokenInfo(type=4 (NEWLINE), string='\n', start=(2, 75), end=(2, 76), line='    Autorzy, którzy tą jednostkę mają wpisani jako AKTUALNA -- czyli"""\n')
TokenInfo(type=0 (ENDMARKER), string='', start=(3, 0), end=(3, 0), line='')
Traceback (most recent call last):
  File "/private/tmp/flake8/c.py", line 13, in <module>
    assert tokens[4].start == (2, 68), tokens[4].start
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: (2, 72)

More info

I found previous similar issue here: #112943

CPython versions tested on:

3.12

Operating systems tested on:

macOS

Linked PRs

Activity

  1. leemars
    added
    type-bugAn unexpected behavior, bug, or error
    on Jun 11, 2024
  2. hugovk
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    on Jun 11, 2024
  3. AlexWaygood
    assigned
    pablogsal
    and
    lysnikolaou
    on Jun 11, 2024

  4. pablogsal commented on Jun 11, 2024

    @pablogsal
    pablogsal
    on Jun 11, 2024
    Member

    Bisected to:

    (memray) ❯ git bisect bad
4a0af0cfdcc0b81da5d78dc219df4985c4403f9c is the first bad commit
commit 4a0af0cfdcc0b81da5d78dc219df4985c4403f9c (HEAD)
Author: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Date:   Tue May 28 22:49:02 2024 +0200

    [3.12] gh-119118: Fix performance regression in tokenize module (GH-119615) (#119683)

    - Cache line object to avoid creating a Unicode object
      for all of the tokens in the same line.
    - Speed up byte offset to column offset conversion by using the
      smallest buffer possible to measure the difference.

    (cherry picked from commit d87b0151062e36e67f9e42e1595fba5bf23a485c)

    Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
    Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

 Misc/NEWS.d/next/Library/2024-05-28-12-15-03.gh-issue-119118.FMKz1F.rst |  2 ++
 Parser/pegen.c                                                          | 25 +++++++++++++++++++++++++
 Parser/pegen.h                                                          |  1 +
 Python/Python-tokenize.c                                                | 44 ++++++++++++++++++++++++++++++++++++++++----
 4 files changed, 68 insertions(+), 4 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Library/2024-05-28-12-15-03.gh-issue-119118.FMKz1F.rst

    @lysnikolaou do you have time to take a look?

  5. lysnikolaou commented on Jun 11, 2024

    @lysnikolaou
    lysnikolaou
    on Jun 11, 2024
    Member

    Yeah, this really seemed like it would be related to that commit. Will have a look in a tiny bit!

  6. pablogsal commented on Jun 11, 2024

    @pablogsal
    pablogsal
    on Jun 11, 2024
    Member

    It seems that is always the f-string end

  7. lysnikolaou
    added 2 commits that reference this issue on Jun 11, 2024

    pythongh-120343: Do not reset byte_col_offset_diff after multiline to…

    a13e65b

    gh-120343: Do not reset byte_col_offset_diff after multiline tokens (#…

    1b62bce
  8. miss-islington
    added 2 commits that reference this issue on Jun 11, 2024

    pythongh-120343: Do not reset byte_col_offset_diff after multiline to…

    098f42c

    pythongh-120343: Do not reset byte_col_offset_diff after multiline to…

    1cb35c0
  9. lysnikolaou
    added 2 commits that reference this issue on Jun 11, 2024

    [3.12] gh-120343: Do not reset byte_col_offset_diff after multiline t…

    0315fdc

    [3.13] gh-120343: Do not reset byte_col_offset_diff after multiline t…

    51bcb67

  10. lysnikolaou commented on Jun 11, 2024

    @lysnikolaou
    lysnikolaou
    on Jun 11, 2024
    Member

    Fixed.

  11. lysnikolaou
    closed this as completedon Jun 11, 2024

  12. 1 remaining item

  13. lysnikolaou
    reopened this on Jun 12, 2024
  14. lysnikolaou
    added 2 commits that reference this issue on Jun 12, 2024

    pythongh-120343: Fix column offsets of multiline tokens in tokenize

    1478261

    gh-120343: Fix column offsets of multiline tokens in tokenize (#120391)

    4b5d3e0
  15. miss-islington
    added 2 commits that reference this issue on Jun 12, 2024

    pythongh-120343: Fix column offsets of multiline tokens in tokenize (p…

    2b8de92

    pythongh-120343: Fix column offsets of multiline tokens in tokenize (p…

    09f776f
  16. lysnikolaou
    added 2 commits that reference this issue on Jun 12, 2024

    [3.12] gh-120343: Fix column offsets of multiline tokens in tokenize (G…

    f75abf8

    [3.13] gh-120343: Fix column offsets of multiline tokens in tokenize (G…

    7c6b342

  17. lysnikolaou commented on Jun 12, 2024

    @lysnikolaou
    lysnikolaou
    on Jun 12, 2024
    Member

    Closing again. Hopefully everything is resolved now. Feel free to reopen in case something weird shows up again.

  18. terryjreedy
    closed this as completedon Jun 15, 2024
  19. mrahtz
    added 2 commits that reference this issue on Jun 30, 2024

    pythongh-120343: Do not reset byte_col_offset_diff after multiline to…

    4901780

    pythongh-120343: Fix column offsets of multiline tokens in tokenize (p…

    d039e36
  20. noahbkim
    added 2 commits that reference this issue on Jul 11, 2024

    pythongh-120343: Do not reset byte_col_offset_diff after multiline to…

    22fe201

    pythongh-120343: Fix column offsets of multiline tokens in tokenize (p…

    642098f
  21. estyxx
    added 2 commits that reference this issue on Jul 17, 2024

    pythongh-120343: Do not reset byte_col_offset_diff after multiline to…

    a7b2029

    pythongh-120343: Fix column offsets of multiline tokens in tokenize (p…

    6cbeb3c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

interpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions