Skip to content

sys.argv is wrong for unicode strings #46381

Closed
Closed
sys.argv is wrong for unicode strings#46381
Labels
OS-windowsinterpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

Description

@GiovanniBajo
GiovanniBajo
mannequin
opened on Feb 16, 2008
BPO 2128
Nosy @loewis, @vstinner, @tiran, @benjaminp
Files
  • argv_unicode.patch
  • wchar.diff
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2008-04-05.20:42:42.184>
created_at = <Date 2008-02-16.16:27:45.539>
labels = ['interpreter-core', 'type-bug', 'OS-windows']
title = 'sys.argv is wrong for unicode strings'
updated_at = <Date 2013-01-14.09:15:31.925>
user = 'https://bugs.python.org/giovannibajo'

    bugs.python.org fields:

    activity = <Date 2013-01-14.09:15:31.925>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2008-04-05.20:42:42.184>
closer = 'loewis'
components = ['Interpreter Core', 'Windows']
creation = <Date 2008-02-16.16:27:45.539>
creator = 'giovannibajo'
dependencies = []
files = ['9449', '9647']
hgrepos = []
issue_num = 2128
keywords = ['patch']
message_count = 15.0
messages = ['62458', '62460', '62499', '62659', '62660', '62664', '63443', '65005', '65045', '65061', '65073', '125827', '125829', '179892', '179928']
nosy_count = 7.0
nosy_names = ['loewis', 'vstinner', 'christian.heimes', 'giovannibajo', 'benjamin.peterson', 'davidsarah', 'mherrmann.at']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue2128'
versions = ['Python 2.6', 'Python 2.5', 'Python 3.0', 'Python 2.7']

    Activity

    1. GiovanniBajo commented on Feb 16, 2008

      @GiovanniBajo
      GiovanniBajomannequin
      on Feb 16, 2008
      MannequinAuthor

      Under Windows, sys.argv is created through the Windows ANSI API.

      When you have a file/directory which can't be represented in the
      system encoding (eg: a japanese-named file or directory on a Western
      Windows), Windows will encode the filename to the system encoding using
      what we call the "replace" policy, and thus sys.argv[] will contain an
      entry like "c:\\foo\\??????????????.dat".

      My suggestion is that:

      • At the Python level, we still expose a single sys.argv[], which will
        contain unicode strings. I think this exactly matches what Py3k does now.

      • At the C level, I believe it involves using GetCommandLineW() and
        CommandLineToArgvW() in WinMain.c, but should Py_Main/PySys_SetArgv() be
        changed to also accept wchar_t** arguments? Or is it better to allow for
        NULL to be passed (under Windows at least), so that the Windows
        code-path in there can use GetCommandLineW()/CommandLineToArgvW() to get
        the current process' arguments?

    2. GiovanniBajo
      added
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      type-bugAn unexpected behavior, bug, or error
      on Feb 16, 2008

    3. tiran commented on Feb 16, 2008

      @tiran
      tiran
      on Feb 16, 2008
      Member

      The issue is related to bpo-1342

      Since we have dropped support for older versions of Windows (9x, ME,
      NT4) I like to get the Python interface to argv, env and files fixed.

    4. tiran
      added
      OS-windows
      on Feb 16, 2008

    5. GiovanniBajo commented on Feb 17, 2008

      @GiovanniBajo
      GiovanniBajomannequin
      on Feb 17, 2008
      MannequinAuthor

      I'm attaching a simple patch that seems to work under Py3k. The trick is
      that Py3k already attempts (not sure how or why) to decode argv using
      utf-8. So it's sufficient to setup argv as UTF8-encoded strings.

      Notice that brings the output of "python ààààà" from this:

      Fatal Python error: no mem for sys.argv
      UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-2:
      invalid data

      to this:

      TypeError: zipimporter() argument 1 must be string without null bytes,
      not str

      which is expected since zipimporter_init() doesn't even know to ignore
      unicode strings (let alone handle them correctly...).

    6. loewis commented on Feb 21, 2008

      @loewis
      loewismannequin
      on Feb 21, 2008
      Mannequin

      I dislike the double decoding, and would prefer if sys.argv would be
      created directly from the wide command line.

      In addition, I think the patch is incorrect: it ignores the arguments to
      Py_Main, which is a documented API function.

      One solution might be to declare all these functions (Py_Main,
      SetProgramName, GetArgcArgv) to operate on Py_UNICODE*, and then
      convert the POSIX callers of Py_Main to use mbstowcs when going
      from the command line to Py_Main. WinMain could then become
      recompiled for Unicode directly, likewise Modules/python.c

    7. GiovanniBajo commented on Feb 21, 2008

      @GiovanniBajo
      GiovanniBajomannequin
      on Feb 21, 2008
      MannequinAuthor

      mbstowcs uses LC_CTYPE. Is that correct and consistent with the way
      default encoding under UNIX is handled by Py3k?

      Would a Py_MainW or similar wrapper be easier on the UNIX guys? I'm just
      asking, I don't have a definite idea.

    8. loewis commented on Feb 21, 2008

      @loewis
      loewismannequin
      on Feb 21, 2008
      Mannequin

      mbstowcs uses LC_CTYPE. Is that correct and consistent with the way
      default encoding under UNIX is handled by Py3k?

      It's correct, but it's not consistent with the default encoding - there
      isn't really any default encoding in Py3k. More specifically,
      PyUnicode_FromString uses UTF-8, but not as a (changeable) default,
      but as part of its API specification.
      Command line arguments are in the locale's charset, so the LC_CTYPE
      must be used to convert them.

      Would a Py_MainW or similar wrapper be easier on the UNIX guys? I'm just
      asking, I don't have a definite idea.

      See above. The current POSIX implementation is incorrect also. It should
      use the locale's encoding, but doesn't.

    9. loewis commented on Mar 10, 2008

      @loewis
      loewismannequin
      on Mar 10, 2008
      Mannequin

      Here is a patch that redoes the entire argv handling, in terms of
      wchar_t. As a side effect, it also changes the sys.path handling to use
      wchar_t.

    10. loewis commented on Apr 5, 2008

      @loewis
      loewismannequin
      on Apr 5, 2008
      Mannequin

      This is now fixed in r62178 for Py3k. For 2.6, I don't think fixing it
      is feasible.

    11. loewis
      closed this as completedon Apr 5, 2008

    12. benjaminp commented on Apr 6, 2008

      @benjaminp
      benjaminp
      on Apr 6, 2008
      Contributor

      MvL's recent commit creates compiler warnings for Unicode UCS4 for the
      same reason as bpo-2388.

    13. loewis commented on Apr 7, 2008

      @loewis
      loewismannequin
      on Apr 7, 2008
      Mannequin

      What warnings precisely are you seeing? I didn't see anything in the 3k
      branch (not even for bpo-2388, as PyErr_Format doesn't have the GCC format
      attribute in 3k, unlike 2.x).

    14. benjaminp commented on Apr 7, 2008

      @benjaminp
      benjaminp
      on Apr 7, 2008
      Contributor

      Martin, you are right that they are not from the same reason as that issue.

      gcc -c -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk/
      -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
      -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/main.o Modules/main.c
      Modules/main.c: In function 'Py_Main':
      Modules/main.c:478: warning: passing argument 1 of 'Py_SetProgramName'
      from incompatible pointer type
      Modules/main.c: In function 'Py_Main':
      Modules/main.c:478: warning: passing argument 1 of 'Py_SetProgramName'
      from incompatible pointer type

    15. davidsarah commented on Jan 9, 2011

      @davidsarah
      davidsarahmannequin
      on Jan 9, 2011
      Mannequin

      The following code is being used to work around this issue for Python 2.x in Tahoe-LAFS:

          # This works around <http://bugs.python.org/issue2128>.
    GetCommandLineW = WINFUNCTYPE(LPWSTR)(("GetCommandLineW", windll.kernel32))
    CommandLineToArgvW = WINFUNCTYPE(POINTER(LPWSTR), LPCWSTR, POINTER(c_int)) \
                            (("CommandLineToArgvW", windll.shell32))

    argc = c_int(0)
    argv_unicode = CommandLineToArgvW(GetCommandLineW(), byref(argc))

    argv = [argv_unicode[i].encode('utf-8') for i in range(0, argc.value)]

    if not hasattr(sys, 'frozen'):
        # If this is an executable produced by py2exe or bbfreeze, then it will
        # have been invoked directly. Otherwise, unicode_argv[0] is the Python
        # interpreter, so skip that.
        argv = argv[1:]

        # Also skip option arguments to the Python interpreter.
        while len(argv) > 0:
            arg = argv[0]
            if not arg.startswith("-") or arg == "-":
                break
            argv = argv[1:]
            if arg == '-m':
                # sys.argv[0] should really be the absolute path of the module source,
                # but never mind
                break
            if arg == '-c':
                argv[0] = '-c'
                break

    16. davidsarah commented on Jan 9, 2011

      @davidsarah
      davidsarahmannequin
      on Jan 9, 2011
      Mannequin

      Sorry, missed out the imports:

          from ctypes import WINFUNCTYPE, windll, POINTER, byref, c_int
    from ctypes.wintypes import LPWSTR, LPCWSTR

    17. mherrmannat commented on Jan 13, 2013

      @mherrmannat
      mherrmannatmannequin
      on Jan 13, 2013
      Mannequin

      Hi,

      is it correct that this bug no longer appears in Python 2.7.3? I checked the changelogs of 2.7, but couldn't find anything.

      Thanks!
      Michael

    18. vstinner commented on Jan 14, 2013

      @vstinner
      vstinner
      on Jan 14, 2013
      Member

      is it correct that this bug no longer appears in Python 2.7.3?

      Martin wrote that it cannot be fixed in Python 2: "For 2.6, I don't think fixing it is feasible."

      The "fix" is to upgrade your application to Python 3.

    19. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      OS-windowsinterpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions