Skip to content

Tokenizer module does not handle backslash characters correctly #90432

Closed
Closed
Tokenizer module does not handle backslash characters correctly#90432
Labels
3.10 (EOL)end of life3.11only security fixes3.12only security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)

Description

@ucodery
ucodery
mannequin
opened on Jan 5, 2022
BPO 46274
Nosy @lysnikolaou, @ucodery

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-01-05.23:12:47.391>
labels = ['interpreter-core', '3.8', '3.9', '3.10', '3.11', '3.7']
title = 'Tokenizer module does not handle backslash characters correctly'
updated_at = <Date 2022-01-06.13:57:33.135>
user = 'https://github.com/ucodery'

bugs.python.org fields:

activity = <Date 2022-01-06.13:57:33.135>
actor = 'pablogsal'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Parser']
creation = <Date 2022-01-05.23:12:47.391>
creator = 'ucodery'
dependencies = []
files = []
hgrepos = []
issue_num = 46274
keywords = []
message_count = 1.0
messages = ['409811']
nosy_count = 2.0
nosy_names = ['lys.nikolaou', 'ucodery']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue46274'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

Activity

  1. ucodery commented on Jan 5, 2022

    @ucodery
    ucoderymannequin
    on Jan 5, 2022
    MannequinAuthor

    A source of one or more backslash-escaped newlines, and one final newline, is not tokenized the same as a source where those lines are "manually joined".

    The source

    \
\
\

    produces the tokens NEWLINE, ENDMARKER when piped to the tokenize module.

    Whereas the source

    produces the tokens NL, ENDMARKER.

    What I expect is to receive only one NL token from both sources. As per the documentation "Two or more physical lines may be joined into logical lines using backslash characters" ... "A logical line that contains only spaces, tabs, formfeeds and possibly a comment, is ignored (i.e., no NEWLINE token is generated)"

    And, because these logical lines are not being ignored, if there are spaces/tabs, INDENT and DEDENT tokens are also being unexpectedly produced.

    The source

        \

    produces the tokens INDENT, NEWLINE, DEDENT, ENDMARKER.

    Whereas the source (with spaces)

    produces the tokens NL, ENDMARKER.

  2. ucodery
    added
    3.7 (EOL)end of life
    3.8 (EOL)end of life
    3.9 (EOL)end of life
    3.10 (EOL)end of life
    3.11only security fixes
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    on Jan 5, 2022
  3. pablogsal
    changed the title [-]backslash creating statement out of nothing[/-] [+]Tokenizer module does not handle backslash characters correctly[/+] on Jan 6, 2022
  4. pablogsal
    changed the title [-]backslash creating statement out of nothing[/-] [+]Tokenizer module does not handle backslash characters correctly[/+] on Jan 6, 2022

  5. 7 remaining items

  6. lysnikolaou commented on May 24, 2023

    @lysnikolaou
    lysnikolaou
    on May 24, 2023
    Member

    This appears to have been fixed in #104323. @asottile Can you check whether that solves the issue for you as well?

  7. asottile commented on May 24, 2023

    @asottile
    asottile
    on May 24, 2023
    Contributor

    @lysnikolaou it seems to fix that -- but also includes a regression(?) (this is 3.12.0b1):

    $ diff -u <(python3.11 -mtokenize t.py) <(python3.12 -mtokenize t.py)
--- /dev/fd/63	2023-05-24 08:44:07.429455783 -0400
+++ /dev/fd/62	2023-05-24 08:44:07.429455783 -0400
@@ -2,7 +2,7 @@
 1,0-1,6:            NAME           'import'       
 1,7-1,10:           NAME           'sys'          
 1,10-1,11:          NEWLINE        '\n'           
-3,0-3,1:            NEWLINE        '\n'           
+3,0-3,1:            NL             '\n'           
 4,0-4,3:            NAME           'def'          
 4,4-4,5:            NAME           'f'            
 4,5-4,6:            OP             '('            
@@ -12,5 +12,5 @@
 5,0-5,4:            INDENT         '    '         
 5,4-5,8:            NAME           'pass'         
 5,8-5,9:            NEWLINE        '\n'           
-6,0-6,0:            DEDENT         ''             
+5,9-5,9:            DEDENT         ''             
 6,0-6,0:            ENDMARKER      ''

  8. lysnikolaou commented on May 24, 2023

    @lysnikolaou
    lysnikolaou
    on May 24, 2023
    Member

    I think that the 3.12 version is the correct one regarding the NEWLINE/NL tokens. Regarding the location of the DEDENT token, I don't have a strong opinion, but feel that it could stay this way. @pablogsal @mgmacias95 What do you think?

  9. pablogsal commented on May 24, 2023

    @pablogsal
    pablogsal
    on May 24, 2023
    Member

    This is now covered in the 3.12 what's new:

    Aditionally, there may be some minor behavioral changes as a consecuence of the changes required to support PEP 701. Some of these changes include:

    Some final DEDENT tokens are now emitted within the bounds of the input. This means that for a file containing 3 lines, the old version of the tokenizer returned a DEDENT token in line 4 whilst the new version returns the token in line 3.
    The type attribute of the tokens emitted when tokenizing some invalid Python characters such as ! has changed from ERRORTOKEN to OP.

  10. pablogsal commented on May 24, 2023

    @pablogsal
    pablogsal
    on May 24, 2023
    Member

    I think we can close this issue

  11. lysnikolaou commented on May 24, 2023

    @lysnikolaou
    lysnikolaou
    on May 24, 2023
    Member

    Thanks @pablogsal!

  12. lysnikolaou
    closed this as completedon May 24, 2023

  13. pablogsal commented on May 25, 2023

    @pablogsal
    pablogsal
    on May 25, 2023
    Member

    Singe the latest fix, the only different now is the DEDENT:

    ❯ diff -u <( ../3.11/python.exe -mtokenize lol.py) <(./python.exe -mtokenize lol.py )
--- /dev/fd/11	2023-05-25 17:08:47
+++ /dev/fd/12	2023-05-25 17:08:47
@@ -12,5 +12,5 @@
 4,0-4,4:            INDENT         '    '
 4,4-4,8:            NAME           'pass'
 4,8-4,9:            NEWLINE        '\n'
-5,0-5,0:            DEDENT         ''
+4,9-4,9:            DEDENT         ''
 5,0-5,0:            ENDMARKER      ''

  14. asottile commented on May 25, 2023

    @asottile
    asottile
    on May 25, 2023
    Contributor

    @pablogsal then this should be reopened

  15. pablogsal commented on May 25, 2023

    @pablogsal
    pablogsal
    on May 25, 2023
    Member

    @pablogsal then this should be reopened

    Why? That is an expected difference as I mentioned in my previous comment.

  16. asottile commented on May 25, 2023

    @asottile
    asottile
    on May 25, 2023
    Contributor

    the NEWLINE token should be a NL if this issue is resolved

  17. pablogsal commented on May 25, 2023

    @pablogsal
    pablogsal
    on May 25, 2023
    Member

    the NEWLINE token should be a NL if this issue is resolved

    No, that was a bug, see #104866 and #104870

  18. asottile commented on May 25, 2023

    @asottile
    asottile
    on May 25, 2023
    Contributor

    I'm confused -- I think we're talking about different things -- are you using the source here?

    #90432 (comment)

  19. pablogsal commented on May 25, 2023

    @pablogsal
    pablogsal
    on May 25, 2023
    Member

    are you using the source here?

    I am. The difference I was mentioning was between Python 3.11 and 3.12. Just to be clear, given

    $cat source1.py
import sys

def f():
    pass

$ cat source2.py
import sys
\

def f():
    pass

    with 3.12:

    ./python -m tokenize < source1.py
1,0-1,6:            NAME           'import'
1,7-1,10:           NAME           'sys'
1,10-1,11:          NEWLINE        '\n'
2,0-2,1:            NL             '\n'
3,0-3,3:            NAME           'def'
3,4-3,5:            NAME           'f'
3,5-3,6:            OP             '('
3,6-3,7:            OP             ')'
3,7-3,8:            OP             ':'
3,8-3,9:            NEWLINE        '\n'
4,0-4,4:            INDENT         '    '
4,4-4,8:            NAME           'pass'
4,8-4,9:            NEWLINE        '\n'
4,9-4,9:            DEDENT         ''
5,0-5,0:            ENDMARKER      ''

./python -m tokenize < source2.py
1,0-1,6:            NAME           'import'
1,7-1,10:           NAME           'sys'
1,10-1,11:          NEWLINE        '\n'
3,0-3,1:            NL             '\n'
4,0-4,3:            NAME           'def'
4,4-4,5:            NAME           'f'
4,5-4,6:            OP             '('
4,6-4,7:            OP             ')'
4,7-4,8:            OP             ':'
4,8-4,9:            NEWLINE        '\n'
5,0-5,4:            INDENT         '    '
5,4-5,8:            NAME           'pass'
5,8-5,9:            NEWLINE        '\n'
5,9-5,9:            DEDENT         ''
6,0-6,0:            ENDMARKER      ''

    With 3.11:

    python -m tokenize < source1.py
1,0-1,6:            NAME           'import'
1,7-1,10:           NAME           'sys'
1,10-1,11:          NEWLINE        '\n'
2,0-2,1:            NL             '\n'
3,0-3,3:            NAME           'def'
3,4-3,5:            NAME           'f'
3,5-3,6:            OP             '('
3,6-3,7:            OP             ')'
3,7-3,8:            OP             ':'
3,8-3,9:            NEWLINE        '\n'
4,0-4,4:            INDENT         '    '
4,4-4,8:            NAME           'pass'
4,8-4,9:            NEWLINE        '\n'
5,0-5,0:            DEDENT         ''
5,0-5,0:            ENDMARKER      ''

python -m tokenize < source2.py
1,0-1,6:            NAME           'import'
1,7-1,10:           NAME           'sys'
1,10-1,11:          NEWLINE        '\n'
3,0-3,1:            NEWLINE        '\n'
4,0-4,3:            NAME           'def'
4,4-4,5:            NAME           'f'
4,5-4,6:            OP             '('
4,6-4,7:            OP             ')'
4,7-4,8:            OP             ':'
4,8-4,9:            NEWLINE        '\n'
5,0-5,4:            INDENT         '    '
5,4-5,8:            NAME           'pass'
5,8-5,9:            NEWLINE        '\n'
6,0-6,0:            DEDENT         ''
6,0-6,0:            ENDMARKER      ''

    The difference between 3.11 and 3.12 for the "correct" version is still the location of the last DEDENT token, which is expected.

  20. asottile commented on May 25, 2023

    @asottile
    asottile
    on May 25, 2023
    Contributor

    hmmm ok but this diff doesn't show the NEWLINE -> NL: #90432 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.10 (EOL)end of life3.11only security fixes3.12only security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions