Repository navigation
3.12: tokenize adds a newline when it is not there #105259
Description
Bug report
tokenize adds a concrete newline character when it is not present. this breaks any sort of roundtrip source generation and pycodestyle's end-of-file checker
here's an example of a file containing a single byte (generated via echo -n 1 > t.py)
# hd -c t.py
00000000 31 |1|
0000000 1
0000001
# python3.12 -m tokenize t.py
0,0-0,0: ENCODING 'utf-8'
1,0-1,1: NUMBER '1'
1,1-1,2: NEWLINE '\n'
2,0-2,0: ENDMARKER ''
# python3.11 -m tokenize t.py
0,0-0,0: ENCODING 'utf-8'
1,0-1,1: NUMBER '1'
1,1-1,2: NEWLINE ''
2,0-2,0: ENDMARKER '' Your environment
- CPython versions tested on: 3.12 dbd7d7c
- Operating system and architecture: ubuntu 22.04 LTS x86_64
cc @pablogsal
Activity
pablogsal commentedon Jun 3, 2023
@lysnikolaou can you take a look at this?
I think this is due to this line:
cpython/Parser/tokenizer.c
Line 1309 in e6373c0
in combination with this line:
cpython/Python/Python-tokenize.c
Line 249 in e6373c0
But the real question is what is going to break if we cover the first line with
if (!tok->extra_tokens)and how we can fix it.~pablogsal commentedon Jun 3, 2023
@lysnikolaou The problem is that we are always adding a new line at the end of the line buffer so we cannot distinguish if the last one is there for real or not (also, we always hardcode
\nforNEWLINEtokens which we should not do for the last one).lysnikolaou commentedon Jun 5, 2023
Do we? I thought you had removed this in #104850, but I guess I misunderstood that PR. I'll have a look.
pablogsal commentedon Jun 5, 2023
That was "reverted" because apparently that was required to keep backward compatibility IIRC. But this is also a different kind of "new line": the tokenize module needs new lines at the end of every logical line to correctly emit the last tokens. Just comment out this part and see what fails (locations of last tokens are wrong as well as the some implicit dedents are missing):
cpython/Parser/tokenizer.c
Line 1309 in e6373c0
Notice that just removing that is not enough to fix this: we would also need to emit the last artificial newline if the input doesn't contain it.
pablogsal commentedon Jun 6, 2023
@Yhg1s can you wait until we fix this for the release?
pythongh-105259: Ensure we don't show newline characters for trailing…
Yhg1s commentedon Jun 6, 2023
Yeah, I can wait.
pythongh-105259: Ensure we don't show newline characters for trailing…
gh-105259: Ensure we don't show newline characters for trailing NEWLI…
pythongh-105259: Ensure we don't show newline characters for trailing…
[3.12] gh-105259: Ensure we don't show newline characters for trailin…
pablogsal commentedon Jun 6, 2023
@asottile can you confirm this fixes it?
asottile commentedon Jun 6, 2023
yep looks fixed -- this is the only outstanding breaking change I'm encountering: #105390