Skip to content

Null characters in strings cause a C SystemError #97556

Closed
Closed
Null characters in strings cause a C SystemError#97556
Labels
3.10 (EOL)end of life3.11only security fixes3.12only security fixes3.9 (EOL)end of lifetype-crashA hard crash of the interpreter, possibly with a core dump

Description

@apccurtiss
apccurtiss
opened on Sep 26, 2022

Crash report

Putting a null byte into a Python string causes a SystemError in Python 3.10, due to a call to strlen in the string parsing library. In Python 3.9, the following example runs without errors:

# -*- coding: latin-1 -*-
"""
<NULL>
"""

In Python 3.10, it raises SystemError: ../Parser/string_parser.c:219: bad argument to internal function.

Internally, the new string_parser library introduced in v3.10.0a1 uses a call to strlen to determine the string size, which is getting thrown off by the null byte. This call is actually unnecessary, as the length has already been calculated by the calling parser and can be retrieved with PyBytes_AsStringAndSize.

Error messages

For single line strings, the error is SystemError: Negative size passed to PyUnicode_New

For multiline strings, the error is SystemError: ../Parser/string_parser.c:219: bad argument to internal function

Linked PRs

Activity

  1. lysnikolaou commented on Apr 25, 2023

    @lysnikolaou
    lysnikolaou
    on Apr 25, 2023
    Member

    I'll have a look at this some time either today or tomorrow during the sprints.

  2. lysnikolaou
    added 2 commits that reference this issue on May 3, 2023

    pythongh-97556: Raise null bytes syntax error upon null in multiline …

    fa7a709

    gh-97556: Raise null bytes syntax error upon null in multiline string (…

    ef0df52

  3. vstinner commented on May 4, 2023

    @vstinner
    vstinner
    on May 4, 2023
    Member

    Script to reproduce the issues:

    with open("script.py", "w") as fp:
    print("# -*- coding: latin-1 -*-", file=fp)
    print('"""', file=fp)
    print('\0', file=fp)
    print('"""', file=fp)

    Current behavior:

    • main branch (3.12): SyntaxError: unterminated triple-quoted string literal (detected at line 3)
    • 3.11: SystemError: Parser/string_parser.c:230: bad argument to internal function

    Oh, it seems like the main branch was just changed (40 min ago) by PR #104136 (just merged).

    @lysnikolaou: Do you consider backporting your change to Python 3.11?

  4. bskinn commented on May 4, 2023

    @bskinn
    bskinn
    on May 4, 2023
    Contributor

    With the merge of #97594 and #104136, I believe what OP has raised is now resolved in main/3.12, and per #96670 (comment) the changes are not to be backported.

    Thus, I think this issue can be closed.

  5. vstinner commented on May 4, 2023

    @vstinner
    vstinner
    on May 4, 2023
    Member

    SystemError: Parser/string_parser.c:230: bad argument to internal function sounds like a bug in Python.

    It's not as if Python 3.11 accepts NUL characters. The question is more about providing a better error message if it finds a NUL character.

  6. bskinn commented on May 4, 2023

    @bskinn
    bskinn
    on May 4, 2023
    Contributor

    It was @gvanrossum's call not to backport the original fix, and I wouldn't presume to speak authoritatively for him. But, I assume the motivation is the xkcd 1172 aspect of changing the error message string.

  7. pablogsal commented on May 4, 2023

    @pablogsal
    pablogsal
    on May 4, 2023
    Member

    I think we should backport this to 3.11

  8. gvanrossum commented on May 4, 2023

    @gvanrossum
    gvanrossum
    on May 4, 2023
    Member

    Okay let’s backport.

  9. bskinn commented on May 4, 2023

    @bskinn
    bskinn
    on May 4, 2023
    Contributor

    Should #97594 be backported also, then?

  10. lysnikolaou commented on May 4, 2023

    @lysnikolaou
    lysnikolaou
    on May 4, 2023
    Member

    Backporting both makes the most sense to me, since just improving the error message would require much more work with the tokenizer buffer working the way it is. If no one has any serious concerns over this, I can work on backporting the necessary PRs.

  11. gvanrossum commented on May 4, 2023

    @gvanrossum
    gvanrossum
    on May 4, 2023
    Member

    Let’s ask @pablogsal about that one.

  12. pablogsal commented on May 4, 2023

    @pablogsal
    pablogsal
    on May 4, 2023
    Member

    Backporting both makes the most sense to me, since just improving the error message would require much more work with the tokenizer buffer working the way it is. If no one has any serious concerns over this, I can work on backporting the necessary PRs.

    Yeah, I prefer to backport the restriction to keep the changes to a minimum in 3.11 as making a bugfix for the error message can be much more complex 👍

  13. lysnikolaou commented on May 7, 2023

    @lysnikolaou
    lysnikolaou
    on May 7, 2023
    Member

    Raising a SyntaxError when parsing NULL bytes has been backported to 3.11. I'm closing this.

  14. lysnikolaou
    closed this as completedon May 7, 2023

  15. vstinner commented on May 9, 2023

    @vstinner
    vstinner
    on May 9, 2023
    Member

    I confirm that 3.11 and main branches now display the same output for ./python script.py (see #97556 (comment) to generate script.py).

      File "/home/vstinner/python/main/../3.11/script.py", line 3
    
SyntaxError: source code cannot contain null bytes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.10 (EOL)end of life3.11only security fixes3.12only security fixes3.9 (EOL)end of lifetype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions