Repository navigation
os.listdir() returns unusable bytes result on Windows #60860
Description
| BPO | 16656 |
|---|---|
| Nosy | @loewis, @amauryfa, @pitrou, @vstinner, @larryhastings, @jkloth, @ezio-melotti, @merwok, @bitdancer, @serhiy-storchaka |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2012-12-16.16:25:00.712>
created_at = <Date 2012-12-10.11:41:44.104>
labels = ['extension-modules', 'type-bug', 'expert-unicode', 'OS-windows']
title = 'os.listdir() returns unusable bytes result on Windows'
updated_at = <Date 2012-12-16.16:25:00.710>
user = 'https://bugs.python.org/techtonik'bugs.python.org fields:
activity = <Date 2012-12-16.16:25:00.710>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2012-12-16.16:25:00.712>
closer = 'pitrou'
components = ['Extension Modules', 'Unicode', 'Windows']
creation = <Date 2012-12-10.11:41:44.104>
creator = 'techtonik'
dependencies = []
files = ['28288', '28289', '28290', '28293', '28305', '28306', '28319']
hgrepos = []
issue_num = 16656
keywords = ['patch']
message_count = 37.0
messages = ['177276', '177279', '177281', '177282', '177283', '177284', '177285', '177331', '177333', '177335', '177339', '177343', '177370', '177373', '177374', '177375', '177377', '177378', '177416', '177438', '177439', '177468', '177472', '177478', '177492', '177494', '177496', '177503', '177508', '177509', '177511', '177512', '177514', '177526', '177534', '177609', '177610']
nosy_count = 11.0
nosy_names = ['loewis', 'amaury.forgeotdarc', 'pitrou', 'vstinner', 'larry', 'techtonik', 'jkloth', 'ezio.melotti', 'eric.araujo', 'r.david.murray', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16656'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4']
Activity
techtonik commentedon Dec 10, 2012
This critical bug is one of the reasons that non-English speaking communities doesn't adopt Python as broadly as it happens in English world compared to other technologies (PHP etc.).
# -- coding: utf-8 --
This gives:
['English name']
[]
Windows Vista.
serhiy-storchaka commentedon Dec 10, 2012
Is it reproduced on 3.x?
bitdancer commentedon Dec 10, 2012
No.
bitdancer commentedon Dec 10, 2012
Oops, clicked submit too soon.
It isn't likely to get fixed in 2.7, because 2.7's unicode support problems is the major reason python3 was developed.
bitdancer commentedon Dec 10, 2012
For that matter, it isn't reproduced in python2.7, either:
jkloth commentedon Dec 10, 2012
The problem exhibited is not coming from the os.walk() implementation, but from the use of a byte-string as the argument to it.
The directories are created with unicode literals and therefore the argument must also be a unicode literal (u'.') for them to be shown. See the note in the listdir() documentation.
As it stands, I suggest that this is closed as invalid, or at the minimum that it could be a documentation bug for walk() not also having a similar note as listdir().
bitdancer commentedon Dec 10, 2012
Works for me without the u'.', too, though less usefully:
Maybe that doesn't work on Windows, though. I am, of course, assuming that python3 does the right thing on Windows, but I can't imagine Victor would have overlooked that.
techtonik commentedon Dec 11, 2012
In Python 3 it fails with UnicodeEncodeError in "C:\Python33\lib\encodings\cp437.py", while Vista's 'dir' command shows everything correctly in the same console, so somebody definitely overlooked that aspect.
This bug is clearly an issue for developers who write products for international markets. It is neither out of date, nor it is invalid. Note in documentation in red is a must have, also a warning should be issued in warning mode when os.walk() ignores international dirs. I doubt there are many people who aware of this racist behavior and want it be default.
amauryfa commentedon Dec 11, 2012
22 remaining items
bitdancer commentedon Dec 14, 2012
Oh, I remember Victor complaining about that behavior of Windows. I'm pretty sure it is the windows API and not python that is doing that mangling. But Victor would know for sure.
larryhastings commentedon Dec 14, 2012
I'm a little confused. FindFirstFile is an ANSI API, so we get a narrow string back. We call PyBytes_FromString(), which expects a narrow string and returns a bytes object. Who's trying (and failing) to encode the filename?
serhiy-storchaka commentedon Dec 14, 2012
Windows. File created using Unicode API and stored UTF-16 encoded in
NTFS. Windows fails to represent this filename using ANSI API.
Here is a patch against 2.7 which always uses Unicode API in listdir()
and tries to encode filenames to str if str argument used.
serhiy-storchaka commentedon Dec 14, 2012
I can't test (and even compile) the patch as I don't have a Windows,
please test it for me.
vstinner commentedon Dec 14, 2012
On Windows with Python 2, unencodable characters are replaced with "?". It is the default behaviour of WideCharToMultiByte() and so all ANSI functions have this behaviour. Python doesn't try to behave differently, it just exposes system function as Python functions.
So for example, os.listdir(bytes) returns filename with "?" if some characters are not encodable to the ANSI codepage. It's a choice in the design of Windows.
I don't understand this point.
PHP doesn't have a Unicode type, I'm quite sure that PHP have exactly the same issue. And this issue is only solved in Python 3... except if you explicitly uses a bytes filename (for os.listdir/os.walk), but the bytes filename API has been deprecated in Python 3.3.
In Python 2, you can use Unicode filenames to workaround this issue. But it doesn't work as well as Python 3: on UNIX, you will get a similar issue with undecodable filenames (which is the opposite of unencodable filenames).
Read my book for more information: https://github.com/haypo/unicode_book/wiki
--
About listdir_unicode-2.7.patch: Python chose to work as Windows with unencodable characters. If you want to change the behaviour, you must change *all* calls to the Windows ANSI API (which is not trivial). Anyway, as I wrote, the bytes API is deprecated for filenames in Python 3.3. I prefer to not change anything in Python 2, because it may break existing applications. For example, os.listdir(bytes) doesn't fail in Python 2.7 with unencodable names, whereas it fails with your patch.
Nothing interesting in this issue, I'm closing it. If your consider the redirection issue important, please open a new issue.
vstinner commentedon Dec 14, 2012
Ooops, I mean: this issue is *already* solved in Python 3
serhiy-storchaka commentedon Dec 14, 2012
No. The purpose of this patch is that it doesn't fail and should return
a usable result.
techtonik commentedon Dec 15, 2012
haypo didn't understood the main user story for this ticket and closed it, so I reopen it with simplified user story.
"""As a developer, I want Python os.listdir('.') function return all directories in current directory on Windows, not matter how international they are, in the same way as other Windows applications return them.
I want returned name to be normal string, but reuseable in subsequent functions, so that I can query stats for this dir or CD to it. It doesn't matter how this name is binary represented. It can be quoted, in UTF-8 byte code or whatever - I don't care as long as I can access the same name from the same session."""
amauryfa commentedon Dec 15, 2012
Anatoly,
print repr(os.listdir(u'.'))
print(ascii(os.listdir('.')))
Do the commands above work correctly?
serhiy-storchaka commentedon Dec 16, 2012
Looks as this is a duplicate of bpo-13247. And Victor submitted a patch with approach similar to me (except that my patch does not raises an exception, but returns an unicode for unencodable names).
It looks as longliving design bug and perhaps really be closed as "won't fix".
pitrou commentedon Dec 16, 2012
Indeed, os.listdir() should always be called with a unicode argument under Windows (which is the natural thing to do in 3.x, anyway). The other issue (with print() unable to display some symbols depending on the codepage) is unrelated.