Repository navigation
imp.find_module() fails with a UnicodeDecodeError when called with non-ASCII search paths #48602
Description
| BPO | 4352 |
|---|---|
| Nosy | @gvanrossum, @amauryfa, @vstinner, @benjaminp, @asvetlov |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2010-10-17.20:37:08.933>
created_at = <Date 2008-11-19.05:17:44.982>
labels = ['interpreter-core', 'type-bug', 'library', 'expert-unicode', 'OS-windows']
title = 'imp.find_module() fails with a UnicodeDecodeError when called with non-ASCII search paths'
updated_at = <Date 2011-12-09.14:55:44.433>
user = 'https://bugs.python.org/JukkaAho'bugs.python.org fields:
activity = <Date 2011-12-09.14:55:44.433>
actor = 'Serg.Asminog'
assignee = 'none'
closed = True
closed_date = <Date 2010-10-17.20:37:08.933>
closer = 'vstinner'
components = ['Interpreter Core', 'Library (Lib)', 'Unicode', 'Windows']
creation = <Date 2008-11-19.05:17:44.982>
creator = 'Jukka Aho'
dependencies = []
files = ['12055', '23891']
hgrepos = []
issue_num = 4352
keywords = []
message_count = 20.0
messages = ['76038', '76045', '76048', '83825', '83828', '83829', '84459', '84512', '84547', '106096', '107833', '108150', '112030', '118981', '149087', '149088', '149097', '149098', '149099', '149102']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'amaury.forgeotdarc', 'vstinner', 'benjamin.peterson', 'Jukka Aho', 'asvetlov', 'Serg.Asminog']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4352'
versions = ['Python 3.0', 'Python 3.1']
Activity
JukkaAho commentedon Nov 19, 2008
imp.find_module() seems to cause an UnicodeDecodeError when the path
list contains paths with non-ASCII names. Tested on Windows [1]; see the
attached test case which demonstrates the problem.
[1] Python 3.0rc2 (r30rc2:67141, Nov 7 2008, 11:43:46) [MSC v.1500 32
bit (Intel)] on win32
[-]imp.find_module() causes UnicodeDecodeError with non-ASCII search paths[/-][+]imp.find_module() fails with a UnicodeDecodeError when called with non-ASCII search paths[/+]vstinner commentedon Nov 19, 2008
The example works correctly on Linux (py3k trunk). The problem is maybe
specific to Windows?
amauryfa commentedon Nov 19, 2008
Indeed. It happens when the filesystem encoding is not utf-8.
I have several changes in my local workspace about this, which also deal
with zipimport and other places that import modules.
I suggest to let 3.0 go out and correct all this for 3.1.
vstinner commentedon Mar 19, 2009
How can I test it on Linux?
vstinner commentedon Mar 19, 2009
Oh, I found sys.setfilesystemencoding("latin-1")! But even with that,
your example find_module.py works correctly with py3k trunk. The
problem has maybe gone?
benjaminp commentedon Mar 19, 2009
Well, latin-1 can decode any arbitrary array of bytes, so of course it
won't fail. :)
asvetlov commentedon Mar 30, 2009
I can reproduce this problem on Windows Vista, fresh py3k sources.
Looks like bug occurs only with Latin-1 characters.
At least Cyrillic works ok.
asvetlov commentedon Mar 30, 2009
From my understanding (after tracing/debugging) problem lies in import.c
find_module tries to convert path from unicode to bytestring using Py_FileSystemDefaultEncoding (line 1397). For Windows it is 'mbcs'.
Conversion done with decode_mbcs (unicodeobject.c:4244) what uses MultiByteToWideChar with codepage CP_ACP. Problem is: converting
composite characters ('\u00e4' is 'a'+'2 dots over letter', I don't know
true name for this sign) this function returns only 'a'.
MSDN says (http://msdn.microsoft.com/en-
us/library/dd374130(VS.85).aspx):
For strings that require validation, such as file, resource, and user
names, the application should always use the WC_NO_BEST_FIT_CHARS flag
with WideCharToMultiByte. This flag prevents the function from mapping
characters to characters that appear similar but have very different
semantics. In some cases, the semantic change can be extreme. For
example, the symbol for "∞" (infinity) maps to 8 (eight) in some code
pages.
Writing encoding function in opposite to PyUnicode_DecodeFSDefault with
setting this flag also cannot help - problematic character just replaced
with 'default' ('?' if not specified).
Hacking specially for 'latin-1' encoding sounds ugly.
Changing all filenames to unicode (with possible usage of fileio instead
of direct calls of open/fdopen) in import.c looks good for me but takes
long time and makes many changes.
gvanrossum commentedon Mar 30, 2009
At the sprint, Andrew Svetlov, Martin von Loewis and I looked into this
a bit, and discovered that Andrew's Vista copy uses a Russian locale for
the filesystem encoding (despite using English as the language). In
this locale, a-umlaut cannot be represented in the ANSI code page (which
has only 256 values), because the Russian locale uses those byte values
to represent Cyrillic.
As long as the import code (written in C) uses bytes in the filesystem
encoding to represent paths, this problem will remain.
Two possible solutions would be to switch to Brett's importlib, or to
change the import code to use wide characters everywhere (like
posixmodule.c). Both are extremely risky and a lot of work, and I don't
expect we'll get to this for 3.1.
(In 2.x the same problem exists, but is perhaps less real because module
names are limited to ASCII.)
We also discovered another problem, which I'll report separately: the
*module* name is decoded to UTF8, while the *path* name uses the
filesystem encoding...
vstinner commentedon May 19, 2010
See also bpo-8611.
vstinner commentedon Jun 14, 2010
About the mbcs encoding: issue bpo-850997 proposes to make it more strict.
vstinner commentedon Jun 18, 2010
I closed issue bpo-850997, mbcs is now really strict by default:
PyUnicode_EncodeFSDefault(), PyUnicode_DecodeFSDefault() and os.fsencode() use mbcs with strict error handler on Windows. On other OS, these functions use surrogateescape error handler, but mbcs only supports strict and replace (to encode, and strict and ignore to decode).
vstinner commentedon Jul 30, 2010
I wrote a patch to fix this issue, see bpo-9425.
vstinner commentedon Oct 17, 2010
Good news: this issue is now fixed in py3k (Python 3.2). I cannot give a commit number, because there are too much commits related to this problem (see bpo-8611 and bpo-9425), but it works ;-)
SergAsminog commentedon Dec 9, 2011
vstinner commentedon Dec 9, 2011
@serg Asminog: What is your Python version? What is your locale encoding (print(sys.getfilesystemencoding())? What is your Windows version?
SergAsminog commentedon Dec 9, 2011
mbcs
nt
3.2.2 (default, Sep 4 2011, 09:07:29) [MSC v.1500 64 bit (AMD64)]
sys.version_info(major=3, minor=2, micro=2, releaselevel='final', serial=0)
win32
-----------
Windows 7 64bit
SergAsminog commentedon Dec 9, 2011
Also
vstinner commentedon Dec 9, 2011
Oops, it's not sys.getfilesystemencoding(), but locale.getpreferredencoding() which is interesting. Can you give me your locale encoding?
SergAsminog commentedon Dec 9, 2011
cp1251