Skip to content

Python startup fails with a fatal error if a command line argument contains an invalid Unicode character #80064

Closed
Closed
Python startup fails with a fatal error if a command line argument contains an invalid Unicode character#80064
Labels
3.10 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifeinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-unicodetype-crashA hard crash of the interpreter, possibly with a core dump

Description

@Neui
Neui
mannequin
opened on Feb 1, 2019
BPO 35883
Nosy @ncoghlan, @vstinner, @ezio-melotti, @eryksun, @miss-islington, @Neui, @jmberg
PRs
  • bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters #24843
  • [3.9] bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (GH-24843) #24905
  • [3.8] bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (GH-24843) #24906
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-10-18.22:15:29.662>
created_at = <Date 2019-02-01.16:55:38.226>
labels = ['interpreter-core', '3.8', 'type-crash', '3.10', 'expert-unicode', '3.9']
title = 'Python startup fails with a fatal error if a command line argument contains an invalid Unicode character'
updated_at = <Date 2021-10-18.22:15:29.662>
user = 'https://github.com/Neui'

    bugs.python.org fields:

    activity = <Date 2021-10-18.22:15:29.662>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-18.22:15:29.662>
closer = 'iritkatriel'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2019-02-01.16:55:38.226>
creator = 'Neui'
dependencies = []
files = []
hgrepos = []
issue_num = 35883
keywords = ['patch']
message_count = 18.0
messages = ['334703', '334705', '334707', '334712', '334732', '369811', '369812', '369813', '369814', '369819', '369820', '388612', '388613', '388614', '388616', '388965', '388970', '389732']
nosy_count = 8.0
nosy_names = ['ncoghlan', 'vstinner', 'ezio.melotti', 'SilentGhost', 'eryksun', 'miss-islington', 'Neui', 'jberg']
pr_nums = ['24843', '24905', '24906']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue35883'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

    Linked PRs

    Activity

    1. jmberg commented on May 24, 2020

      @jmberg
      jmbergmannequin
      on May 24, 2020
      Mannequin

      In fact that python one-liner works with just about everything else that you can throw at it, just not something that "looks like utf-8 but isn't".

      And of course adding LC_CTYPE=ascii or something like that fixes it, as you'd expect. Then the "surrogateescape" works fine, since mbstowcs() won't try to decode it as utf-8.

    2. jmberg commented on May 24, 2020

      @jmberg
      jmbergmannequin
      on May 24, 2020
      Mannequin

      And wrt. _Py_DecodeUTF8Ex() - it doesn't seem to help. But that's probably because I'm not __ANDROID__, nor __APPLE__, and then regardless of current_locale being non-zero or not, we end up in decode_current_locale() where the impedance mismatch happens.

      Setting PYTHONUTF8=1 in the environment works too, in that case we do get into _Py_DecodeUTF8Ex().

    3. jmberg commented on May 24, 2020

      @jmberg
      jmbergmannequin
      on May 24, 2020
      Mannequin

      Like I said above, it could be argued that the bug is in glibc, and then

      https://p.sipsolutions.net/6a4e9fce82dbbfa0.txt

      could be used as a simple LD_PRELOAD wrapper to work around this, just to illustrate the problem from that side.

      Arguably, that makes glibc in violation of RFC 3629, since it says:

      1. UTF-8 definition

      [...]

      In UTF-8, characters from the U+0000..U+10FFFF range (the UTF-16
      accessible range) are encoded using sequences of 1 to 4 octets.

      [...]

        (hexadecimal)    |              (binary)

      --------------------+---------------------------------------------
      0000 0000-0000 007F | 0xxxxxxx
      0000 0080-0000 07FF | 110xxxxx 10xxxxxx
      0000 0800-0000 FFFF | 1110xxxx 10xxxxxx 10xxxxxx
      0001 0000-0010 FFFF | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

      [...]

      Implementations of the decoding algorithm above MUST protect against
      decoding invalid sequences.

      [...]

      Here's a simple test program:

      https://p.sipsolutions.net/ac091b4ea4b7f742.txt

    4. jmberg commented on May 24, 2020

      @jmberg
      jmbergmannequin
      on May 24, 2020
      Mannequin

      I've also filed https://sourceware.org/bugzilla/show_bug.cgi?id=26034 for glibc, because that's where really the issues seems to be?

      But perhaps python should be forgiving of glibc errors here.

    5. eryksun
      added
      topic-unicode
      and removed
      3.7 (EOL)end of life
      on Mar 12, 2021

    6. vstinner commented on Mar 13, 2021

      @vstinner
      vstinner
      on Mar 13, 2021
      Member

      I wrote PR 24843 to fix this issue. With this fix, os.fsencode(sys.argv[1]) returns the original byte sequence as expected.

      --

      I dislike the replace error handler since it loses information. The PEP-383 surrogateescape error handler exists to prevent losing information.

      The root issue is that Py_DecodeLocale() creates wide characters outside Python Unicode valid range: [U+0000; U+10ffff].

      On Linux, Py_DecodeLocale() usually calls mbstowcs() of the C library. The problem is that the the glibc UTF-8 decoder doesn't respect the RFC 3629, it doesn't reject characters outside [U+0000; U+10ffff] range. The following issue requests to change the glibc UTF-8 codec to respect the RFC 3629, but it's open since 2006:
      https://sourceware.org/bugzilla/show_bug.cgi?id=2373

      Even if the glibc changes, Python should behave the same on old glibc version.

      My PEP modifies Py_DecodeLocale() to check if there are characters outside [U+0000; U+10ffff] range and use the surrogateescape error handler in that case.

    7. vstinner commented on Mar 13, 2021

      @vstinner
      vstinner
      on Mar 13, 2021
      Member

      https://bugs.python.org/issue25631 "Segmentation fault with invalid Unicode command-line arguments in embedded Python" (actually 'fixed' since it now abort()s)

      This issue is different: it is about the Py_Main() function called explicitly when Python is embedded in an application. Python fails if the command line contains a *wide character* outside the [U+0000; U+10ffff] range.

      This issue is about Python on Linux in which case Py_BytesMain() is used to decode *bytes* from the command line.

    8. vstinner
      changed the title [-]Change invalid unicode characters to replacement characters in argv[/-] [+]Python startup fails with a fatal error if a command line argument contains an invalid Unicode character[/+] on Mar 13, 2021

    9. vstinner commented on Mar 13, 2021

      @vstinner
      vstinner
      on Mar 13, 2021
      Member

      This shouldn't be an issue in 3.7, at least not with the default UTF-8 mode configuration. With this mode, Py_DecodeLocale calls _Py_DecodeUTF8Ex using the surrogateescape error handler [1].

      Right, enabling explicitly the Python UTF-8 Mode works around the issue:
      https://docs.python.org/dev/library/os.html#python-utf-8-mode

      $ python3.10 -c 'import sys; print(ascii(sys.argv))' $'\U7fffbeba'
Fatal Python error: init_interp_main: failed to update the Python config
Python runtime state: core initialized
ValueError: character U+7fffbeba is not in range [U+0000; U+10ffff]

      Current thread 0x00007effa1891740 (most recent call first):
      <no Python frame>

      $ python3.10 -X utf8 -c 'import sys; print(ascii(sys.argv))' $'\U7fffbeba'
['-c', '\udcfd\udcbf\udcbf\udcbb\udcba\udcba']

    10. vstinner commented on Mar 13, 2021

      @vstinner
      vstinner
      on Mar 13, 2021
      Member

      Right, enabling explicitly the Python UTF-8 Mode works around the issue

      When the Python UTF-8 Mode is used, on macOS or on Android, Python uses its own UTF-8 decoder which respects the RFC 3629: it rejects characters outside [U+0000; U+10ffff].

      Otherwise, Python relies on the libc mbstowcs() decoder which may or may not create characters outside the [U+0000; U+10ffff] range. I understand that this issue is mostly about the UTF-8 encoding, I don't think that other encodings can produce characters greater than U+10ffff code point.

    11. vstinner commented on Mar 17, 2021

      @vstinner
      vstinner
      on Mar 17, 2021
      Member

      New changeset 9976834 by Victor Stinner in branch 'master':
      bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (GH-24843)
      9976834

    12. miss-islington commented on Mar 17, 2021

      @miss-islington
      miss-islington
      on Mar 17, 2021
      Contributor

      New changeset aa967ec by Miss Islington (bot) in branch '3.9':
      bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (GH-24843)
      aa967ec

    13. vstinner commented on Mar 29, 2021

      @vstinner
      vstinner
      on Mar 29, 2021
      Member

      New changeset 3b6e61e by Miss Islington (bot) in branch '3.8':
      bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (GH-24843) (GH-24906)
      3b6e61e

    14. iritkatriel
      closed this as completedon Oct 18, 2021
    15. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    16. vstinner
      added 2 commits that reference this issue on May 30, 2023

      pythongh-80064: Fix is_valid_wide_char() return type

      da953ea

      gh-80064: Fix is_valid_wide_char() return type (#105099)

      4b65d56
    17. carljm
      added a commit that references this issue on May 30, 2023

      Merge branch 'main' into superbadgetattro

      24e1159
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifeinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-unicodetype-crashA hard crash of the interpreter, possibly with a core dump

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions