sys.argv is wrong for unicode strings

@loewis

BPO	2128
Nosy	@loewis, @vstinner, @tiran, @benjaminp
Files	argv_unicode.patch wchar.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2008-04-05.20:42:42.184>
created_at = <Date 2008-02-16.16:27:45.539>
labels = ['interpreter-core', 'type-bug', 'OS-windows']
title = 'sys.argv is wrong for unicode strings'
updated_at = <Date 2013-01-14.09:15:31.925>
user = 'https://bugs.python.org/giovannibajo'

bugs.python.org fields:

activity = <Date 2013-01-14.09:15:31.925>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2008-04-05.20:42:42.184>
closer = 'loewis'
components = ['Interpreter Core', 'Windows']
creation = <Date 2008-02-16.16:27:45.539>
creator = 'giovannibajo'
dependencies = []
files = ['9449', '9647']
hgrepos = []
issue_num = 2128
keywords = ['patch']
message_count = 15.0
messages = ['62458', '62460', '62499', '62659', '62660', '62664', '63443', '65005', '65045', '65061', '65073', '125827', '125829', '179892', '179928']
nosy_count = 7.0
nosy_names = ['loewis', 'vstinner', 'christian.heimes', 'giovannibajo', 'benjamin.peterson', 'davidsarah', 'mherrmann.at']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue2128'
versions = ['Python 2.6', 'Python 2.5', 'Python 3.0', 'Python 2.7']

Under Windows, sys.argv is created through the Windows ANSI API.

When you have a file/directory which can't be represented in the
system encoding (eg: a japanese-named file or directory on a Western
Windows), Windows will encode the filename to the system encoding using
what we call the "replace" policy, and thus sys.argv[] will contain an
entry like "c:\\foo\\??????????????.dat".

My suggestion is that:

At the Python level, we still expose a single sys.argv[], which will
contain unicode strings. I think this exactly matches what Py3k does now.
At the C level, I believe it involves using GetCommandLineW() and
CommandLineToArgvW() in WinMain.c, but should Py_Main/PySys_SetArgv() be
changed to also accept wchar_t** arguments? Or is it better to allow for
NULL to be passed (under Windows at least), so that the Windows
code-path in there can use GetCommandLineW()/CommandLineToArgvW() to get
the current process' arguments?

The issue is related to bpo-1342

Since we have dropped support for older versions of Windows (9x, ME,
NT4) I like to get the Python interface to argv, env and files fixed.

I'm attaching a simple patch that seems to work under Py3k. The trick is
that Py3k already attempts (not sure how or why) to decode argv using
utf-8. So it's sufficient to setup argv as UTF8-encoded strings.

Notice that brings the output of "python ààààà" from this:

Fatal Python error: no mem for sys.argv
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-2:
invalid data

to this:

TypeError: zipimporter() argument 1 must be string without null bytes,
not str

which is expected since zipimporter_init() doesn't even know to ignore
unicode strings (let alone handle them correctly...).

I dislike the double decoding, and would prefer if sys.argv would be
created directly from the wide command line.

In addition, I think the patch is incorrect: it ignores the arguments to
Py_Main, which is a documented API function.

One solution might be to declare all these functions (Py_Main,
SetProgramName, GetArgcArgv) to operate on Py_UNICODE*, and then
convert the POSIX callers of Py_Main to use mbstowcs when going
from the command line to Py_Main. WinMain could then become
recompiled for Unicode directly, likewise Modules/python.c

mbstowcs uses LC_CTYPE. Is that correct and consistent with the way
default encoding under UNIX is handled by Py3k?

Would a Py_MainW or similar wrapper be easier on the UNIX guys? I'm just
asking, I don't have a definite idea.

mbstowcs uses LC_CTYPE. Is that correct and consistent with the way
default encoding under UNIX is handled by Py3k?

It's correct, but it's not consistent with the default encoding - there
isn't really any default encoding in Py3k. More specifically,
PyUnicode_FromString uses UTF-8, but not as a (changeable) default,
but as part of its API specification.
Command line arguments are in the locale's charset, so the LC_CTYPE
must be used to convert them.

Would a Py_MainW or similar wrapper be easier on the UNIX guys? I'm just
asking, I don't have a definite idea.

See above. The current POSIX implementation is incorrect also. It should
use the locale's encoding, but doesn't.

Here is a patch that redoes the entire argv handling, in terms of
wchar_t. As a side effect, it also changes the sys.path handling to use
wchar_t.

This is now fixed in r62178 for Py3k. For 2.6, I don't think fixing it
is feasible.

MvL's recent commit creates compiler warnings for Unicode UCS4 for the
same reason as bpo-2388.

What warnings precisely are you seeing? I didn't see anything in the 3k
branch (not even for bpo-2388, as PyErr_Format doesn't have the GCC format
attribute in 3k, unlike 2.x).

Martin, you are right that they are not from the same reason as that issue.

gcc -c -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk/
-fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
-I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/main.o Modules/main.c
Modules/main.c: In function 'Py_Main':
Modules/main.c:478: warning: passing argument 1 of 'Py_SetProgramName'
from incompatible pointer type
Modules/main.c: In function 'Py_Main':
Modules/main.c:478: warning: passing argument 1 of 'Py_SetProgramName'
from incompatible pointer type

The following code is being used to work around this issue for Python 2.x in Tahoe-LAFS:

    # This works around <http://bugs.python.org/issue2128>.
    GetCommandLineW = WINFUNCTYPE(LPWSTR)(("GetCommandLineW", windll.kernel32))
    CommandLineToArgvW = WINFUNCTYPE(POINTER(LPWSTR), LPCWSTR, POINTER(c_int)) \
                            (("CommandLineToArgvW", windll.shell32))

    argc = c_int(0)
    argv_unicode = CommandLineToArgvW(GetCommandLineW(), byref(argc))

    argv = [argv_unicode[i].encode('utf-8') for i in range(0, argc.value)]

    if not hasattr(sys, 'frozen'):
        # If this is an executable produced by py2exe or bbfreeze, then it will
        # have been invoked directly. Otherwise, unicode_argv[0] is the Python
        # interpreter, so skip that.
        argv = argv[1:]

        # Also skip option arguments to the Python interpreter.
        while len(argv) > 0:
            arg = argv[0]
            if not arg.startswith("-") or arg == "-":
                break
            argv = argv[1:]
            if arg == '-m':
                # sys.argv[0] should really be the absolute path of the module source,
                # but never mind
                break
            if arg == '-c':
                argv[0] = '-c'
                break

Sorry, missed out the imports:

    from ctypes import WINFUNCTYPE, windll, POINTER, byref, c_int
    from ctypes.wintypes import LPWSTR, LPCWSTR

Hi,

is it correct that this bug no longer appears in Python 2.7.3? I checked the changelogs of 2.7, but couldn't find anything.

Thanks!
Michael

is it correct that this bug no longer appears in Python 2.7.3?

Martin wrote that it cannot be fixed in Python 2: "For 2.6, I don't think fixing it is feasible."

The "fix" is to upgrade your application to Python 3.

Navigation Menu

Repository navigation

sys.argv is wrong for unicode strings #46381

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

sys.argv is wrong for unicode strings #46381

Description

Activity

GiovanniBajo commented on Feb 16, 2008

tiran commented on Feb 16, 2008

GiovanniBajo commented on Feb 17, 2008

loewis commented on Feb 21, 2008

GiovanniBajo commented on Feb 21, 2008

loewis commented on Feb 21, 2008

loewis commented on Mar 10, 2008

loewis commented on Apr 5, 2008

benjaminp commented on Apr 6, 2008

loewis commented on Apr 7, 2008

benjaminp commented on Apr 7, 2008

davidsarah commented on Jan 9, 2011

davidsarah commented on Jan 9, 2011

mherrmannat commented on Jan 13, 2013

vstinner commented on Jan 14, 2013

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions