Repository navigation
Add fsencode() functions to os module #52760
Description
| BPO | 8514 |
|---|---|
| Nosy | @malemburg, @loewis, @gpshead, @pitrou, @vstinner, @benjaminp, @ezio-melotti |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2010-05-08.11:12:37.545>
created_at = <Date 2010-04-23.23:39:10.536>
labels = ['type-feature', 'library', 'expert-unicode']
title = 'Add fsencode() functions to os module'
updated_at = <Date 2010-05-09.03:18:11.154>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2010-05-09.03:18:11.154>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2010-05-08.11:12:37.545>
closer = 'vstinner'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2010-04-23.23:39:10.536>
creator = 'vstinner'
dependencies = []
files = ['17241']
hgrepos = []
issue_num = 8514
keywords = ['patch']
message_count = 36.0
messages = ['104063', '104064', '104068', '104147', '104185', '104186', '104200', '104210', '104214', '104218', '104220', '104224', '104225', '104236', '104635', '104648', '104650', '104652', '104654', '104672', '104723', '104802', '104823', '104826', '104869', '104874', '104876', '104896', '104921', '105171', '105264', '105278', '105301', '105363', '105364', '105368']
nosy_count = 8.0
nosy_names = ['lemburg', 'loewis', 'gregory.p.smith', 'pitrou', 'vstinner', 'benjamin.peterson', 'ezio.melotti', 'Arfrever']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue8514'
versions = ['Python 3.2']
Activity
vstinner commentedon Apr 23, 2010
Python3 uses unicode filenames in Windows and bytes filenames (but support also unicode filenames) on other OS. We have to support both types. On POSIX system, bytes filenames can be stored in unicode filenames using sys.getfilesystemencoding() and the surrogateescape error handler (to store undecodable bytes as unicode surrogates, see PEP-383).
I would like to create fs_encode() and fs_decode() in os.path to ease the manipulation of filenames in the two bytes (str and bytes).
On Windows, fs_decode() and fs_encode() don't touch the filename, but reject filenames of types different than str (unicode) with a TypeError, especially bytes filename.
Mac OS X rejects invalid UTF-8 filenames, and so surrogateescape should maybe not be used on this OS.
Attached patch is an implementation of this issue.
vstinner commentedon Apr 23, 2010
Issue bpo-8513 would benefit from these functions.
malemburg commentedon Apr 24, 2010
STINNER Victor wrote:
Please follow the naming convention used in os.path. The functions
would have to be called os.path.fsencode() and os.path.fsdecode().
Other than that, I'm +0 on the patch: the sys.filesystemencoding logic
doesn't really work well in practice - on Unix and BSD platforms, there's
no such thing as a single system-wide file system and consequently,
the file system encoding depends on the path you are looking at. For most
of those file systems, the name is just a sequence of bytes with arbitrary
encoding.
vstinner commentedon Apr 25, 2010
Ok
Today, most POSIX system uses utf8 by default for all partitions. If you mount an USB key, CD-Rom or network shared directory with the wrong options, you may get filenames in a different encoding. But this issue is not about fixing your OS configuration, but helping the most common case: a system using the same encoding everywhere (for the whole file system).
You are still free to use directly the native OS type (unicode on Windows, bytes on other OS), ie. don't use fsencode()/fsdecode().
Python3 prefers unicode, eg. print expects an unicode string, not a byte string. I mean it's more pratical to use unicode everywhere in Python, and so fsencode()/fsdecode() can be really useful on POSIX systems.
vstinner commentedon Apr 25, 2010
Update path: rename fs_encode/fs_decode to fsencode/fsdecode.
[-]Create fs_encode() and fs_decode() functions in os.path[/-][+]Create fsencode() and fsdecode() functions in os.path[/+]vstinner commentedon Apr 25, 2010
Oops, "Update path": I mean "Update patch" ;-)
gpshead commentedon Apr 26, 2010
i'm +0.7 on fsencode/fsdecode going into os.path.
My bikeshed 0.7? They're also useful for dealing with environment variables which are not strictly filesystem (fs) related but also suffer from the same issue requiring surrogate escape. But other than just calling these os.encode and os.decode I don't have any brilliant alternate naming suggestions. thoughts? I could easily live with os.path.fsencode/fsdecode, I just wanted to point the other use out.
vstinner commentedon Apr 26, 2010
Yes, Python3 decodes environment variables using sys.getfilesystemencoding()+surrogateescape. And since my last fix on os.execve(), subprocess (and os.execv(p)e) uses also surrogateescape to encode environment variables.
And yes again, I also patched os.getenv() to decode bytes name to unicode using sys.getfilesystemencoding()+surrogateescape.
*fs*encode() and *fs*decode() is a reference to the encoding: sys.get*filesystem*encoding().
See also issue bpo-8513.
vstinner commentedon Apr 26, 2010
Oh! In Python3, ntpath.expanduser() supports bytes path and uses sys.getfilesystemencoding() to encode an unicode environment variable to a byte string.
Should we remove bytes path support in ntpath.expanduser(), or support bytes in ntpath.fsencode()/.fsdecode()?
(sys.getfilesystemencoding() is "mbcs" on Windows)
malemburg commentedon Apr 26, 2010
STINNER Victor wrote:
I don't see what environment variables have to do with the file
system.
Those are two different contexts and thus also require two different
approaches to the problem.
Command line parameters are another area, where an encoding
comes into play, but this again does not have to coincide with the
file system encoding.
Also note that "mbcs" on Windows is a meta-encoding. The
implementation of that encoding depends on the locale used by
the Windows user. It's just a coincidence that this may actually
work for the environment variables on Windows as well, but there's
no guarantee.
On Unix, you often have the case that the environment variables
use mixed encodings, e.g. the CGI interface is a good example
where this happens per definition. The CGI environment can
includes file system paths, data encoded in Latin-1 (or some
other encoding), etc.
See http://www.ietf.org/rfc/rfc3875.txt for details.
Environment variables are also commonly used to interface
to external programs from daemons, e.g. postfix, procmail
and others use environment variables to communicate with
external helper applications.
malemburg commentedon Apr 26, 2010
STINNER Victor wrote:
Right, but if you start using those new API in standard lib
functions, programmers no longer have that choice.
In real life applications, you do run into these problems quite
often, so instead of coding against an ideal world, we have to be
aware of the problems and make it possible for the standard lib modules
to deal with them.
Sure, but forcing UnicodeDecodeErrors upon Python3 programmers is
not a good idea. Please keep that in mind.
Thanks,
Marc-Andre Lemburg
eGenix.com
::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
15 remaining items
loewis commentedon May 3, 2010
Why is that? In msg104063, you claim that you want to create these
functions to deal with file names (not environment variables), in
msg104064, you claim that bpo-8513 (which is about the program name in
subprocess) would benefit from these functions. Do these use cases
become invalid if os.environb becomes available?
vstinner commentedon May 3, 2010
Yes, but my os_path_fs_encode_decode-3.patch uses it in getenv() which is maybe a bad idea: os.environb may avoid this.
bpo-8513 is also related to environment variables: subprocess._execute_child() calls os.get_exec_path() which search the PATH environment variable. It would be nice to support bytes environment variable in the env argument of Popen constructor (bytes key and/or value).
loewis commentedon May 3, 2010
STINNER Victor wrote:
IIUC, that usage is an equivalent transformation, i.e. the code doesn't
change its behavior. It is mere refactorization.
So *if* these functions are accepted, this change is a good idea
regardless of the os.environb introduction (unless I'm missing
something, and there is indeed a behavior change).
I still fail to see why this would make this issue block on the
os.environb introduction. Whether this gets introduced or not, the
program name issue remains, no?
vstinner commentedon May 3, 2010
I changed os.getenv() to accept byte string key (in a previous commit), but I don't like this hack. If we have os.environb, os.getenv() shouldn't support bytes anymore (but use str only, as before).
--
I worked a little more on fsencode()/os.environb, trying to fix all issues. fsdecode() is no more needed if we have os.environb, and fsencode() can be simplified to:
fsdecode() leads to mojibake.
vstinner commentedon May 4, 2010
I think that fsencode() (and fsdecode()) should be specific to POSIX. I don't know any good reason to encode a nice and correctly encoded unicode string to the ugly MBCS "encoding".
vstinner commentedon May 6, 2010
New short, simple and clean path: add os.fsencode() for Unix only.
--
Don't create it for Windows to encourage the usage of unicode on Windows (and use MBCS is a bad idea). fsdecode() was a also bad idea: it's better to keep bytes unchanged on Unix, and it's now possible thanks to os.environb and os.getenvb().
gpshead commentedon May 8, 2010
+.. function:: fsencode(value)
+
+ Encode *value* to bytes for use in the file system, environment variables or
+ the command line. Use :func:`sys.getfilesystemencoding` and
+ ``'surrogateescape'`` error handler for str, and keep bytes unchanged.
I'd word the latter sentence as:
Uses :func:`sys.getfilesystemencoding` and ``'surrogateescape'`` error handler for strings and returns bytes unchanged.
Otherwise I think this patch looks good. +1
[-]Create fsencode() and fsdecode() functions in os.path[/-][+]Add fsencode() functions to os module[/+]vstinner commentedon May 8, 2010
Commited: r80971 (py3k), blocked by r80972 (3.1).
benjaminp commentedon May 8, 2010
Why does this have no tests?
vstinner commentedon May 9, 2010
The function is trivial. Does it really need tests? What kind of tests?
fsencode() is already tested indirectly by test_subprocess, and bpo-8513 will add
new tests.
benjaminp commentedon May 9, 2010
2010/5/8 STINNER Victor <report@bugs.python.org>:
Check that it is equivalent to utf-8 with surrogatesescape then.
Excuses, excuses!
vstinner commentedon May 9, 2010
The file system encoding can be anything, not only utf-8. Anyway: r81014.