Repository navigation
Command-line arguments are not correctly decoded if locale and fileystem encodings are different #54201
Description
| BPO | 9992 |
|---|---|
| Nosy | @malemburg, @loewis, @ronaldoussoren, @pitrou, @vstinner, @pjenvey, @merwok |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/vstinner'
closed_at = <Date 2010-10-14.10:56:13.499>
created_at = <Date 2010-09-29.22:36:21.301>
labels = ['interpreter-core', 'expert-unicode']
title = 'Command-line arguments are not correctly decoded if locale and fileystem encodings are different'
updated_at = <Date 2010-10-14.10:59:27.140>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2010-10-14.10:59:27.140>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2010-10-14.10:56:13.499>
closer = 'vstinner'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2010-09-29.22:36:21.301>
creator = 'vstinner'
dependencies = []
files = ['19062', '19064', '19184', '19190']
hgrepos = []
issue_num = 9992
keywords = ['patch']
message_count = 49.0
messages = ['117669', '117676', '117705', '117709', '117711', '117716', '117717', '117871', '118221', '118225', '118257', '118258', '118263', '118264', '118268', '118269', '118270', '118271', '118278', '118279', '118336', '118337', '118339', '118340', '118341', '118344', '118352', '118358', '118359', '118360', '118365', '118367', '118368', '118374', '118375', '118377', '118385', '118386', '118388', '118389', '118390', '118392', '118394', '118591', '118607', '118633', '118645', '118647', '118648']
nosy_count = 8.0
nosy_names = ['lemburg', 'loewis', 'ixokai', 'ronaldoussoren', 'pitrou', 'vstinner', 'pjenvey', 'eric.araujo']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue9992'
versions = ['Python 3.2']
Activity
vstinner commentedon Sep 29, 2010
On UNIX/BSD systems, Python decodes arguments with the locale encoding, whereas subprocess encodes arguments with the fileystem encoding. If both encodings are differents, we have a problem.
There was already the issue bpo-4388 but it was closed because it was specific to old versions of Mac OS X. With the PYTHONFSENCODING environment variable (added to Python 3.2), it is easy to trigger this issue: run Python with a filesystem encoding different than the locale encoding. Attached script demonstrates the bug.
--
I see two possible encodings to encode and decode command line arguments (with surrogateescape error handler):
(a) filesystem encoding
(b) locale encoding
Decode Python command line arguments is one of the first operation executed when running Python, in the main() function. We don't have import machinery or codec API available at this moment. So I don't see how we can use the filesystem encoding here. Read issue bpo-9630 to see how complex it is to use the filesystem encoding when initializing Python.
Use the locale encoding is easier because we already have _Py_char2wchar() and _Py_wchar2char() functions to decode/encode with the locale encoding and the surrogateescape error handler. These functions use the wchar_t* type which is less pratical than PyUnicodeObject*, but it is an advantage because wchar_t* type doesn't need Python to be completly initialized (whereas some PyUnicode methods loads modules, eg. encode and decode).
In bpo-8775, I proposed to create a new variable to store the "command line encoding": sys.getcmdlineencoding(). But this issue was closed because there was only one use case: bpo-4388 (which was closed but not fixed).
I don't know, or don't really care, how sys.getcmdlineencoding() should be initialized. The important point is that we have to use the same encoding to decode and encode command line arguments.
--
I don't really know if using another encoding is the right solution. The problem is maybe that the filesystem encoding should not be controlable by the user?
And what about environment variables: should we continue to encode and decode them with the filesystem encoding, or should we use the new "command line encoding"?
vstinner commentedon Sep 29, 2010
[cmdline_encoding-2.patch] Patch to use locale encoding to decode and encode command line arguments. Remarks about the patch:
The patch is an updated version of the patch attached to bpo-8775.
Using the patch, locale_fs_encoding.py test script pass.
malemburg commentedon Sep 30, 2010
STINNER Victor wrote:
The problem with command line arguments is that they don't necessarily
have just one encoding (just like env vars may well use more than
one encoding) on Unix platforms.
When using path and file names on the command line they will likely
use the file system encoding. When passing in configuration variables,
the arguments will likely use the current locale settings.
The use of wchar C lib functions is not ideal for parsing the
command line arguments, since this always uses the locale
settings.
Creating a copy as Python3 of argv is also not ideal,
since manipulating argv to change the OS process ps-output is
common on Unix, and there is currently no access (AFAIK) provided
to the original argv array passed to Python in Python3.
I think we should use a similar approach as the one for os.environ
here, where we keep the original bytes buffers around and have
a second copy with str objects which may not necessarily be
complete (e.g. when decoding a string fails).
Unfortunately, the use of wchar_t for command line arguments
has already spread throughout the code base, so I see little
chance of fixing this use.
What we could do, is at least make the original bytes version
of argv available to Python, so that decoding errors can be worked
around in the application (just like we have for os.environ with
os.environb).
[-]Command line arguments are not correctly decoded if locale and fileystem encodings are different[/-][+]Command line arguments are not correctly decoded if locale and fileystem encodings are different[/+]vstinner commentedon Sep 30, 2010
The issue bpo-8776 proposes the creation of sys.argv.
Ok, and? We have to pick up one and use it. We cannot guess the encoding of
each argument, nor change sys.argv to use bytes. (And the creation sys.argvb
will not solve this issue.)
I still think that using the filesystem encoding is not possible for technical
reasons (it might be possible, but it will be very hard), whereas I attached a
working patch to use the locale encoding.
[-]Command line arguments are not correctly decoded if locale and fileystem encodings are different[/-][+]Command line arguments are not correctly decodedif locale and fileystem encodings aredifferent[/+]malemburg commentedon Sep 30, 2010
STINNER Victor wrote:
Right, I think you meant sys.argvb and yes, I think it's a good idea.
Sure and using the locale setting is fine. The point is that we pick
one, but keep the original data around for the application to use in
case it knows better, so this will solve the problem.
[-]Command line arguments are not correctly decodedif locale and fileystem encodings aredifferent[/-][+]Command line arguments are not correctly decodedif locale and fileystem encodings aredifferent[/+]vstinner commentedon Sep 30, 2010
Extract of an interesting message (msg111432) of bpo-8775 (issue specific to Mac OS X):
<< A system where the filesystem encoding doesn't match the locale encoding is hard to get right. While it would be possible to add sys.cmdlineencoding that doesn't actually solve the semantic problem because external tools might not cooperate.
That is, most system tools seem to work with bytes internally and do not treat arguments as text encoded in the locale encoding that should be re-encoded in the filesystem encoding before passing them to the C APIs.
That is, when calling "ls somefile" the "ls" command will pass the bytes in argv[1] to the POSIX routines for getting file information without trying to reencode. >>
vstinner commentedon Sep 30, 2010
Mmmh. The problem is maybe that the new PYTHONFSENCODING environment variable (added by bpo-8622) introduced an horrible inconstency between Python and other applications. Other applications ignore PYTHONFSENCODING.
The simplest solution to fix this issue is to remove PYTHONFSENCODING variable. In this case, the user have to set LANG, LC_ALL or LC_CTYPE, instead of PYTHONFSENCODING, to set Python filesystem encoding.
vstinner commentedon Oct 2, 2010
See also bpo-10014: sys.path[0] is decoded from the locale encoding instead of the fileystem encoding.
ixokai commentedon Oct 8, 2010
This issue seems to be the cause of bpo-4388 -- and cmdline_encoding-2.patch fixes it, fwiw.
pitrou commentedon Oct 8, 2010
I don't think I agree with this. It's only important when you run a Python interpreter using subprocess, but the point of using subprocess is to run something *else* than Python. This something else generally expects filenames in their correct bytes representation, not in a mojibaked version hand-tuned for Python.
34 remaining items
loewis commentedon Oct 11, 2010
If the user hasn't switched to UTF-8, why would Python need that
to properly decode file names?
vstinner commentedon Oct 11, 2010
MAL> You can't just tell people to go with whatever encoding setup
MAL> you prefer to make Python's guessing easier or more correct.
Python doesn't really *guess* the encoding, it just reads the encoding from the locale.
What do you mean by "more correct"? How can Python knowns the right encoding better than the user? Python should not guess anything. If the environment is not correctly configured, it's not Python's fault. The user has to fix its environment.
vstinner commentedon Oct 11, 2010
That's why there are different environement variables:
vstinner commentedon Oct 11, 2010
bpo-9992.patch:
$ diffstat issue9992.patch Doc/using/cmdline.rst | 12Doc/whatsnew/3.2.rst | 6 ------
Lib/test/test_os.py | 30 ------------------------------
Lib/test/test_subprocess.py | 4 ----
Lib/test/test_sys.py | 29 -----------------------------
Modules/main.c | 3 ---
Modules/python.c | 10 +++++++++-
Python/pythonrun.c | 22 ++++++----------------
8 files changed, 15 insertions(+), 101 deletions(-)
I like such patch: it removes more code than it adds, but it fixes 4 different issues!
I didn't tested the patch specific to OSX (use utf8 to decode command line arguments).
vstinner commentedon Oct 11, 2010
I think that bpo-9992.patch fixes also bpo-4388 because it uses the same encoding (FS encoding, utf8) on OSX to encode and to decode command line arguments.
[-]Command line arguments are not correctly decodediflocale and fileystem encodingsaredifferent[/-][+]Command-line arguments are not correctly decoded if locale and fileystem encodings are different[/+]vstinner commentedon Oct 13, 2010
I commited bpo-9992.patch as r85430 (remove PYTHONFSENCODING) + r85435 (OSX: decode command line arguments from utf-8).
These commits should fix this issue. Reopen the issue if you notice new problems, or if the problem is not fixed yet. I will watch Mac OS X buildbots, especially about r85435 ;-)
vstinner commentedon Oct 14, 2010
test_undecodable_env() of test_subprocess fails. r85430 removes the following code which was added by Antoine to fix this issue.
# Force surrogate-escaping of \xFF in the child process;
# otherwise it can be decoded as-is if the default locale
# is latin-1.
env['PYTHONFSENCODING'] = 'ascii'
I think that we should accept that b'\xff' can be decoded as '\xff' and that's all.
pitrou commentedon Oct 14, 2010
What do you plan to do to fix this failure?
======================================================================
FAIL: test_undecodable_env (test.test_subprocess.POSIXProcessTestCase)
----------------------------------------------------------------------
http://www.python.org/dev/buildbot/builders/x86%20debian%20parallel%203.x/builds/502/steps/test/logs/stdio
vstinner commentedon Oct 14, 2010
With r85466+r85467, the test_undecodable_env (of test_subprocess) uses C locale to get ASCII locale encoding (for the first test, on unicode environment variables). It should have the same effect than env['PYTHONFSENCODING'] = 'ascii': get ASCII as the filesystem encoding.
vstinner commentedon Oct 14, 2010
Ok, the issue is not complelty fixed ;-)
12:55 < py-bb> build #504 of x86 debian parallel 3.x is complete: Success [build successful] Build details are at
http://www.python.org/dev/buildbot/all/builders/x86%20debian%20parallel%203.x/builds/504
vstinner commentedon Oct 14, 2010
I tried... "the issue is *now* complelty fixed"