Repository navigation
Token lines in the tokenize module always end in new line #104825
Description
Activity
Yeah, I ran across that while working on #104798. Are you already working on a fix or shall I take it up?
#104846 now removes a newline, even when a newline exists in the original source.
❯ ./python.exe Python 3.13.0a0 (heads/setuptools-peg-generator-dirty:c7503656c2, May 24 2023, 12:24:20) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import tokenize >>> import io >>> file = io.StringIO('1 + 2 + 3\n') >>> for token in tokenize.generate_tokens(file.readline): ... print(token) ... TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1 + 2 + 3') TokenInfo(type=55 (OP), string='+', start=(1, 2), end=(1, 3), line='1 + 2 + 3') TokenInfo(type=2 (NUMBER), string='2', start=(1, 4), end=(1, 5), line='1 + 2 + 3') TokenInfo(type=55 (OP), string='+', start=(1, 6), end=(1, 7), line='1 + 2 + 3') TokenInfo(type=2 (NUMBER), string='3', start=(1, 8), end=(1, 9), line='1 + 2 + 3') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 9), end=(1, 10), line='1 + 2 + 3') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')
- added a commit that references this issue
on May 24, 2023 @pablogsal Do you think the above should be fixed? Will you work on it?
@pablogsal Do you think the above should be fixed? Will you work on it?
Hummmmm, not sure how easy is going to be for us to distinguish between implicit and explicit newlines at the end. I would say, leave it like is now unless you disagree, in which case let's see if we can fix it easily.
My only concern is that tests that rely on that will fail. For example,
test_pegennow fails:def test_repeat_0_simple(self) -> None: grammar = """ start: thing thing* NEWLINE thing: NUMBER """ parser_class = make_parser(grammar) node = parse_string("1 2 3\n", parser_class) self.assertEqual( node, [ TokenInfo(NUMBER, string="1", start=(1, 0), end=(1, 1), line="1 2 3\n"), [ TokenInfo( NUMBER, string="2", start=(1, 2), end=(1, 3), line="1 2 3\n" ), TokenInfo( NUMBER, string="3", start=(1, 4), end=(1, 5), line="1 2 3\n" ), ], TokenInfo( NEWLINE, string="\n", start=(1, 5), end=(1, 6), line="1 2 3\n" ), ], ) node = parse_string("1\n", parser_class) self.assertEqual( node, [ TokenInfo(NUMBER, string="1", start=(1, 0), end=(1, 1), line="1\n"), [], TokenInfo(NEWLINE, string="\n", start=(1, 1), end=(1, 2), line="1\n"), ], )
The failure is:
====================================================================== FAIL: test_repeat_0_simple (test.test_peg_generator.test_pegen.TestPegen.test_repeat_0_simple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/lysnikolaou/repos/python/cpython/Lib/test/test_peg_generator/test_pegen.py", line 248, in test_repeat_0_simple self.assertEqual( AssertionError: Lists differ: [Toke[67 chars]1 2 3'), [TokenInfo(type=2 (NUMBER), string='2[201 chars] 3')] != [Toke[67 chars]1 2 3\n'), [TokenInfo(type=2 (NUMBER), string=[209 chars]\n')] First differing element 0: TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1 2 3') TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1 2 3\n') Diff is 1018 characters long. Set self.maxDiff to None to see it.The fix is very simple, so I'm okay with going with this as-is. I will fix
test_pegenin my setuptools PR.- added a commit that references this issue
on May 24, 2023 - added a commit that references this issue
on May 24, 2023 You say:
As a consequence of the changes in the Python tokenizer post PEP701, now the line attribute always ends in a new line character. We should remove this as it adds no information and wasn't there before.
But it looks to me like the lines used to always end in newline, except for ENDMARKER. This uses tokbug.py from #104972:
3.10.9 (main, Dec 7 2022, 07:15:24) [Clang 12.0.0 (clang-1200.0.32.29)] TokenInfo(type=62 (NL), string='\n', start=(1, 0), end=(1, 1), line='\n') TokenInfo(type=1 (NAME), string='a', start=(2, 0), end=(2, 1), line='a + \\\n') TokenInfo(type=54 (OP), string='+', start=(2, 2), end=(2, 3), line='a + \\\n') TokenInfo(type=1 (NAME), string='b', start=(3, 0), end=(3, 1), line='b\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(3, 1), end=(3, 2), line='b\n') TokenInfo(type=0 (ENDMARKER), string='', start=(4, 0), end=(4, 0), line='')
As a consequence of the changes in the Python tokenizer post PEP701, now the
lineattribute always ends in a new line character. We should remove this as it adds no information and wasn't there before.Linked PRs