Repository navigation
under Windows, os.path.abspath returns non-ASCII bytes paths as question marks #57456
Description
| BPO | 13247 |
|---|---|
| Nosy | @loewis, @terryjreedy, @atsuoishimoto, @vstinner, @tjguk, @zware, @serhiy-storchaka, @zooba |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2014-06-22.23:29:17.370>
created_at = <Date 2011-10-22.23:45:23.895>
labels = ['invalid', 'type-bug', 'library', 'OS-windows']
title = 'under Windows, os.path.abspath returns non-ASCII bytes paths as question marks'
updated_at = <Date 2014-06-22.23:29:17.369>
user = 'https://bugs.python.org/ubershmekel'bugs.python.org fields:
activity = <Date 2014-06-22.23:29:17.369>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2014-06-22.23:29:17.370>
closer = 'vstinner'
components = ['Library (Lib)', 'Windows']
creation = <Date 2011-10-22.23:45:23.895>
creator = 'ubershmekel'
dependencies = []
files = ['23521']
hgrepos = []
issue_num = 13247
keywords = ['patch']
message_count = 23.0
messages = ['146204', '146222', '146322', '146396', '146397', '146403', '146405', '146406', '146407', '146418', '146422', '146424', '146427', '146428', '146451', '146462', '146465', '177612', '220006', '220157', '221203', '221226', '221325']
nosy_count = 11.0
nosy_names = ['loewis', 'terry.reedy', 'ishimoto', 'vstinner', 'tim.golden', 'ubershmekel', 'BreamoreBoy', 'python-dev', 'zach.ware', 'serhiy.storchaka', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13247'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4']
Activity
ubershmekel commentedon Oct 22, 2011
For Python 2:
Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on win32 >>> os.path.abspath('.') 'C:\\Users\\yuv\\Desktop\\YuvDesktop\\??????' >>> os.path.abspath(u'.') u'C:\\Users\\yuv\\Desktop\\YuvDesktop\\\u05d0\u05d1\u05d2\u05d3\u05d4\u05d5' For Python 3: Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit (Intel)] on win32 >>> os.path.abspath('.') 'C:\\Users\\yuv\\Desktop\\YuvDesktop\\\u05d0\u05d1\u05d2\u05d3\u05d4\u05d5' >>> os.path.abspath(b'.') b'C:\\Users\\yuv\\Desktop\\YuvDesktop\\??????'The returned path with question marks is completely useless. It's better that python throw an error than return the question marks. Another option is to try and get the ascii version of the path, I believe windows has one.
vstinner commentedon Oct 23, 2011
abspath() is implemented using nt._getfullpathname() which calls GetFullPathNameA().
Can you open the file using such filename? If no, I agree that the result is useless.
Python is currently a thin wrapper on the Windows API. Windows doesn't consider that a filename with question marks as an error.
http://msdn.microsoft.com/en-us/library/windows/desktop/aa364963%28v=vs.85%29.aspx
Python can maybe uses GetFullPathNameW() and encode manually the filename using its strict MBCS codec. MBCS codec is strict since Python 3.2: it raises a UnicodeEncodeError if the string cannot be encoded.
ubershmekel commentedon Oct 24, 2011
An example error with abspath and bytes input:
>>> os.path.abspath('.') 'C:\\Users\\yuv\\Desktop\\YuvDesktop\\\u05d0\u05d1\u05d2\u05d3\u05d4\u05d5' >>> os.path.abspath(b'.') b'C:\\Users\\yuv\\Desktop\\YuvDesktop\\??????' >>> os.listdir(os.path.abspath(b'.')) Traceback (most recent call last): File "<stdin>", line 1, in <module> WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: 'C:\\Users\\yuv\\Desktop\\YuvDesktop\\??????/*.*' >>>I couldn't follow the implementation, I got stuck not being able to locate the definition for os.getcwdb so I couldn't join you for that part. Here's another possible solution:
>>> win32api.GetFullPathName('.') 'C:\\Users\\yuv\\Desktop\\YuvDesktop\\\u05d0\u05d1\u05d2\u05d3\u05d4\u05d5' >>> win32api.GetShortPathName(win32api.GetFullPathName('.')) 'C:\\Users\\yuv\\Desktop\\YUVDES~1\\5F30~1'The short path is ascii but the problem is not all windows file systems have 8.3 filenames [1]. So I think your suggestion is the best solution.
[1] http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#short_vs.\_long_names
vstinner commentedon Oct 25, 2011
os.getcwdb() (GetCurrentDirectoryA) and os.listdir(bytes) (FindNextFileA & co) encode filenames using WideCharToMultiByte() in default mode (flags=0): unencodable characters are replaced by question marks. Such filenames cannot be used, open() fails with OSError(22, "invalid argument: '?'") for example.
Attached patch changes os.getcwdb() and os.listdir(bytes) to use the Windows native API (wide character API) with Python MBCS codec in strict mode (error handler "strict") to notify directly the user that the filename cannot be decoded.
The patch only changes the behaviour for filename not encodable to the ANSI code page, such filenames are rare.
vstinner commentedon Oct 25, 2011
os_mbcs.patch adds _Py_EncodeCodePage() to encode directly wchar_t* filenames without having to create a temporary Unicode object.
The patch removes HAVE_MBCS because the MBCS is now always needed by the posixmodule.c. Anyway, I don't see why MultiByteToWideChar() and WideCharToMultiByte() would not be available on Windows.
atsuoishimoto commentedon Oct 25, 2011
-1 from me.
I hate to see Unicode exceptions here. It would be an another source of mysterious Unicode exception. Programmers and users would be confused by error message. If you make such characters error, Python should raise an OSError or such.
File names with '?' are fine to display informations to users. Not all file names are nessesary to be used to open files.
I don't think filenames cannot be decoded in ANSI code page are rare enough to be ignored. I use Japanese edition of windows, but I sometime receive files with Chinese or German names.
Or, in some case, I have to change codepage with 'chcp 437' command to run console application made for American environment. I seldom run such application in these days, though.
vstinner commentedon Oct 26, 2011
Le 26/10/2011 01:32, Atsuo Ishimoto a écrit :
The issue is able being able to be noticied of encoding errors.
Currently, unencodable characters are silently replaced and you don't
know if the filename is valid or not. If a UnicodeEncodeError is raised,
you will be noticed and so you have to fix the problem.
Anyway, you must use the Unicode API on Windows. If you use the Unicode
API, filenames are no more encoded and code pages are no more used, so
bye bye Unicode errors!
The Windows bytes API is just kept for backward compatibility. More
details in my email to python-dev:
http://mail.python.org/pipermail/python-dev/2011-October/114203.html
terryjreedy commentedon Oct 26, 2011
The doc says "All functions accepting path or file names accept both bytes and string objects, and result in an object of the same type, if a path or file name is returned." It does that now (the encoding assumed or produced for bytes is not specified). It says nothing about raising exceptions in certain situations. So this is a feature change request, one that would likely break existing code.
Users can test for invalid returned paths with "'?' in returned_path", though I admit that the use of '?' as a glob, regex, and url special char makes it a bad choice of error char.
atsuoishimoto commentedon Oct 26, 2011
On Wed, Oct 26, 2011 at 9:12 AM, STINNER Victor <report@bugs.python.org> wrote:
This patch solve nothing, but just raises exception. It can break
existing codes. Also, I don't think it worth to add weired behavior to
Python std lib. I'll be surprised if *Byte* API raised an
UnicodeEncodeError.
Agreed. So I would like to suggest not to adding unnecessary
complexity to the Byte API.
ubershmekel commentedon Oct 26, 2011
I use python a lot with Hebrew and many websites have internationalization which may involve unicode paths. I agree that saying "unicode paths are rare" is inaccurate.
If the current situation isn't fixed though - you just can't use the resulting path for almost anything. Do you have a use case Ishimoto?
Windows XP and up implement paths as unicode, that means that a bytes api doesn't even make sense unless python does some encoding and decoding for you. E.g. python can use the unicode API's internally and return utf-8 encoded bytes. But you couldn't use these paths outside of python. The fact is you shouldn't be doing os.path.abspath(b'.') in windows to begin with.
ubershmekel commentedon Oct 26, 2011
Another option btw is to use utf-16, which will work but it's a bit ugly as well:
Tested on python 2.7, but you know what I mean.
6 remaining items
vstinner commentedon Oct 26, 2011
Oops, I specified the wrong issue number in my changeset 2cad20e2e588, it's the issue bpo-13216.
[-]os.path.abspath returns unicode paths as question marks[/-][+]under Windows, os.path.abspath returns non-ASCII bytes paths as question marks[/+]serhiy-storchaka commentedon Dec 16, 2012
See also bpo-16656 where another approach was proposed (unicode names returned from Bytes API if result is not encodable). Actually I think now that there is no right solution of this issue.
BreamoreBoy commentedon Jun 8, 2014
I've read this entire issue and can't see that much can be done, so suggest it is closed as "won't fix" as has already happened for bpo-16656, which was suggested is a duplicate of this in msg177609.
vstinner commentedon Jun 10, 2014
My patch can be applied in Python 3.5 to notice immediatly users that filenames cannot be encoded to the ANSI code page. Anyway, bytes filenames are deprecated (emit a DeprecationWarning warning) in the os module on Windows since Python 3.3.
BreamoreBoy commentedon Jun 21, 2014
Can someone do a patch review please, it's way over my head, and set the stage and versions as appropriate.
loewis commentedon Jun 22, 2014
I'm -1 on the patch. The string currently returned might be useless, but the fundamental problem is that using bytes for filenames on Windows just isn't sufficient for all cases. Microsoft has chosen to return question marks in the API, and Python should return them as the system vendor did.
Another alternative would be to switch to UTF-8 as the file system encoding on Windows, but that change might be too incompatible.
vstinner commentedon Jun 22, 2014
Ok to keep calls to ANSI versions of the Windows API when bytes filenames are used (so get question marks on encoding errors).
On Linux, I tried to have more than one "OS" encoding and it was a big fail (search for "PYTHONFSENCODING" env var in Python history). It introduced many new tricky issues. In short, Python should use the same "OS encoding" *everyone*. Since they are many places where Python doesn't control the encoding, we must use the same encoding than the OS. For example, os.listdir(b'.') uses the ANSI code page. If you concatenate two strings, one encoding to UTF-8 and the other encoded to the ANSI code page, you will at least see mojibake, and your operation will probably fail (ex: unable to open the file).
I mean that forcing an encoding *everywhere* is a losing battle. There are too many external functions using the locale encoding on UNIX and the ANSI code page on Windows. Not only in the C library, think also to OpenSSL just to give you one example.
Anyway, bytes filenames are deprecated since Python 3.2 so it's maybe time to stop using them!
--
Another alternative is to completly drop support of bytes filenames on Windows in Python 3.5. But I expect that too many applications will just fail. It's too early for such disruptive change.
So I'm just closing the issue as "not a bug", because Python just follows the vendor choice (Microsoft decided to use funny question marks :-)).