Repository navigation
mishandling of c-strings in parser #96670
Description
Bug report
the parser mishandles lines containing null bytes when parsing source -- this allows the code to be misleadingly different from what it looks like.
I've been told by security@ that it is ok to post this publicly.
in the below example, <NUL> is an actual null byte:
x = '<NUL>' nothing to see here
';import os;os.system('echo pwnd')and the execution and appearance in the terminal:
$ cat t.py
x = '' nothing to see here
';import os;os.system('echo pwnd')
$ python3 t.py
pwndit appears that after splitting the source into lines, the individual lines are treated as c strings and so the null terminator is misinterpreted, jamming the string contents together and it executes similar to this:
x = '';import os;os.system('echo pwnd')note that if you want to write out a file like this here's a simple bit of code you can paste into an interactive prompt:
open('t.py', 'w').write("x = '\0' nothing to see here\n';import os;os.system('echo pwnd')\n")here is perhaps a shorter example:
open('t.py', 'w').write("x = 1\0 + 1\n+2\nprint(x)\n")I originally found this due to a bug report where the ast parser rejects code containing null bytes:
>>> import ast
>>> ast.parse("x = '\0'")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/ast.py", line 47, in parse
return compile(source, filename, mode, flags,
ValueError: source code string cannot contain null bytes
>>> ast.parse(b"x = '\0'")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/ast.py", line 47, in parse
return compile(source, filename, mode, flags,
ValueError: source code string cannot contain null bytesideally I would want the interpreter to reject files containing null bytes as a SyntaxError (and update the ast.parse error to a SyntaxError as well) -- though it appears there are some of these files in the wild -- such as https://github.com/univention/univention-corporate-server/blob/5.0-2/services/univention-ldb-modules/buildtools/bin/waf-svn
Your environment
- CPython versions tested on: 3.7 ... 3.11rc1 (though pretty sure this reproduces on all versions)
- Operating system and architecture: ubuntu 22.04, linux, x86_64
Activity
encukou commentedon Sep 8, 2022
There was a discussion about the larger issue of code being ”misleadingly different from what it looks like", and the general consensus was that this should be solved in editors, linters and code review tools, rather than in Python. IMO it was this thread but might need more digging.
That doesn't mean handling of
\0can't be changed. Or of control characters in general: see the informational PEP 672 for some more details.Perhaps all unusual control characters should be banned? Even in strings? With a warning for 2 releases, according to the backwards compatibility policy – waf would need to switch to e.g.
base64.asottile commentedon Sep 8, 2022
that was the rationale for making this public, but it's different than those as it's a mishandling by the parser rather than a quirk of how unicode displays
ypankovych commentedon Sep 9, 2022
Ugh, this is freakin amazing.
And run it:
open('t.py', 'w').write("\0'''\0How do i print \'Hello World\' in Python?\n\0Here is how:\nprint(\'Hello World\')\0'''")gpshead commentedon Sep 13, 2022
FWIW I'd be in favor of that behavior change as a Feature. I wouldn't backport it as a bug though.
asottile commentedon Sep 13, 2022
there's kinda 3 things I think:
ValueError=>SyntaxErrorinast.parse(probably 3.12+)SyntaxError(probably also 3.12+)(char*, size_t)or whatever function (probably backportable?)does it make sense to pursue these three things and separately?
gvanrossum commentedon Sep 13, 2022
Agreed, it's changing behavior, so can't backport. I like SyntaxError for this situation. The root exception isn't in ast.parse(), it's in compile(), probably even deeper.
That's the biggest hole, should probably do this first.
Depends on where that fix is (can you tell I haven't looked at the source code yet? :-). If any of the affected functions are public it's going to be more difficult.
Likely.
gpshead commentedon Sep 14, 2022
Be careful about this one. Python pre-pended to raw binary data for the executed Python to locate within the file and use (embedded zip or other data) is a common idiom that must continue to work.
gvanrossum commentedon Sep 14, 2022
Are you sure that works? Unlike Unix shells, Python parses the entire source file before executing any code. How would you get Python to ignore a blob of arbitrary binary data embedded in the source code, even if
\0is accepted? If the blob contains\ncharacters you can't hide it behind a#comment. I suppose you could prefix it with a"""quote, if you can arrange for the file to also end in""", and you're lucky that the blob doesn't contain embedded"""sequences.But if I had to do something like that I'd probably just embed the Python code in a bash script as a "here" document and end the bash script with
exit.gpshead commentedon Sep 14, 2022
Oh you're right, I guess what I've seen do that is a bash+python+data hybrid monster.
gvanrossum commentedon Sep 14, 2022
Okay then we should be safe banning
\0in files starting with 3.12.asottile commentedon Sep 14, 2022
I suspect one could probably hack something together like this unfortunately
gvanrossum commentedon Sep 14, 2022
You'd still have to arrange for the actual binary not to contain the sequence
'''-- because that would end the string started atGARB =. I don't see how setting the coding to Latin-1 changes matters. (Honestly it seems you're making the same mistake as Greg.)asottile commentedon Sep 14, 2022
latin1 is to prevent a decoding error while parsing the source -- I linked an in-the-wild example of such a file in the original post
gvanrossum commentedon Sep 14, 2022
Okay, that's impressive. It looks like the blob is lightly encoded --
\nand\rare encoded using#.and#&. (I'm curious what they'd do if the blob contains one of those sequences, there doesn't seem to be a way to quote them.)The proposed change in 3.12 will break them, but they have version checks and they can just cope with it, I don't think we need to preserve this machine-dependent quirk forever. The code looks like it had to deal with various other versioning issues already (e.g. it tries to handle Python 2 and 3!).
10 remaining items
fixup! pythongh-96670: Raise SyntaxError when parsing NULL bytes
fixup! fixup! pythongh-96670: Raise SyntaxError when parsing NULL bytes
pythongh-96670: Raise SyntaxError when parsing NULL bytes
pythongh-96670: Raise SyntaxError when parsing NULL bytes
pythongh-96670: Raise SyntaxError when parsing NULL bytes
pythongh-96670: Raise SyntaxError when parsing NULL bytes
pythongh-96670: Raise SyntaxError when parsing NULL bytes
pythongh-96670: Raise SyntaxError when parsing NULL bytes
gh-96670: Raise SyntaxError when parsing NULL bytes (#97594)
gvanrossum commentedon Sep 27, 2022
Closing this. Starting with Python 3.12, NUL bytes won't be allowed in source code read from files. We're not backporting this since this could be considered a feature by some.
vstinner commentedon Sep 28, 2022
It was changed by the commit aab01e3: PR #97594.
CarliJoy commentedon Oct 17, 2022
Just for reference: The problem is already mentioned in https://peps.python.org/pep-0672/#control-characters
And Pylint checks for null characters already.
[3.11] pythongh-96670: Raise SyntaxError when parsing NULL bytes (pyt…
[3.11] gh-96670: Raise SyntaxError when parsing NULL bytes (GH-97594) (…