Repository navigation
Rewrite import machinery to work with unicode paths #53671
Description
| BPO | 9425 |
|---|---|
| Nosy | @amauryfa, @pitrou, @kristjanvalur, @vstinner, @ezio-melotti, @merwok, @florentx |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2010-10-19.01:02:36.457>
created_at = <Date 2010-07-30.00:13:32.930>
labels = ['interpreter-core', 'expert-unicode']
title = 'Rewrite import machinery to work with unicode paths'
updated_at = <Date 2010-10-19.01:02:36.455>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2010-10-19.01:02:36.455>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2010-10-19.01:02:36.457>
closer = 'vstinner'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2010-07-30.00:13:32.930>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 9425
keywords = ['patch', 'buildbot']
message_count = 58.0
messages = ['112026', '112027', '112032', '112038', '112039', '112213', '113164', '113165', '113255', '113256', '113259', '113261', '113308', '113342', '113347', '113351', '113353', '113354', '113355', '113546', '113548', '113598', '113726', '113761', '113764', '113771', '113785', '113795', '113796', '113834', '113835', '113843', '113852', '113855', '113859', '113862', '113903', '113904', '113913', '113915', '113955', '113956', '114002', '114059', '114062', '114078', '114080', '114087', '114089', '114090', '114192', '114819', '114827', '114944', '115180', '115343', '117630', '119097']
nosy_count = 9.0
nosy_names = ['amaury.forgeotdarc', 'pitrou', 'kristjan.jonsson', 'vstinner', 'ezio.melotti', 'eric.araujo', 'Arfrever', 'flox', 'Romme']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue9425'
versions = ['Python 3.2']
Activity
vstinner commentedon Jul 30, 2010
Python (2 and 3) is unable to load a module installed in a directory containing characters not encodable to the locale encoding. And Python doesn't work if it's installed in non-ASCII directory on Windows or with a locale encoding different than UTF-8. On Windows, the locale encoding is "mbcs", which is a small charset, unable to mix different languages, whereas the file system is fully unicode compatible (it uses UTF-16). Python should work with unicode strings (wchar_t*, Py_UNICODE* or PyUnicodeObject) instead of byte strings (char* or PyBytesObject), especially while loading a Python module.
It's not an easy task because it requires to change a lot of code, especially in Python/import.c. I am working on this topic since some months and I have now a working patch. It's now possible to run Python from the source tree containing a non-ASCII character in C locale (ASCII encoding). Except just a minor bug in test_gdb, all tests of the test suite pass.
I posted the whole patch on Rietveld for a review:
http://codereview.appspot.com/1874048
The patch is huge because it fixes different things:
a) import machinery (import.c, getpath.c, importdl.c, ...)
b) many error handlers using filenames (compile.c, errors.c, _warnings.c, sysmodule.c, ...)
c) functions using filenames, especially Python full path: log the filename (eg. Lib/distutils/file_util.py), filename written to a program output (eg. Lib/platform.py)
d) tests (Lib/test/test_*.py)
(b), (c) and (d) can be fixed before/without (a). But (a) requires other parts to work correctly.
If it's not possible to review the patch, I can try to split it in smaller parts.
--
Related issues:
bpo-3080: Full unicode import system
bpo-4352: imp.find_module() fails with a UnicodeDecodeError
when called with non-ASCII search paths
bpo-8611: Python3 doesn't support locale different than utf8
and an non-ASCII path (POSIX)
bpo-8988: import + coding = failure (3.1.2/win32)
--
See also my email sent to python-dev for more information:
http://mail.python.org/pipermail/python-dev/2010-July/101619.html
vstinner commentedon Jul 30, 2010
Oh, I forgot to say that I created an svn branch including my work: import_unicode.
http://svn.python.org/view/python/branches/import_unicode/
You can try it if you prefer svn to an huge patch.
I created a branch so you can follow my work commit by commit using svn history.
--
The patch is not completly done. There are still remaining FIXMEs. Some FIXME are not bugs, but improvments. The most important FIXME is to restore the support of bytes path in sys.path. I removed it temporary, because it was easier for me.
ezio-melotti commentedon Jul 30, 2010
I wrote a few minor comments on codereview.
The patch should also include more tests.
vstinner commentedon Jul 30, 2010
Which kind of test? Run the test suite in a non-ASCII directory with encoding different than utf-8 is enough. If the patch is accepted, the solution is maybe a specific buildbot.
vstinner commentedon Jul 30, 2010
Another important TODO: use weak references for the code objects list.
--
I tested my patch on Windows. I fixes bpo-8988 because non-ASCII characters are now correctly decoded with mbcs and not UTF-8. But it doesn't work with characters not encodable to mbcs. It looks like there are some remaining code using byte string. I fixed some of them in import_unicode branch, but it's not enough.
It is not easy to investigate because Visual Studio refuse to compile the project if the project directory contains a character not encodable to mbcs. And it is unable to debug python if the project directory is renamed after the compilation. I will maybe retry with Cygwin or with the old school "printf" method.
It looks like few Windows applications support characters not encodable to mbcs (locale encoding): MinGW and WinSCP do neither support such characters.
vstinner commentedon Jul 31, 2010
After some tests on Windows, I realized that my patch is not enough to be fully unicode compliant (on Windows). Some functions are still using PyUnicode_DecodeFSDefault() or PyUnicode_EncodeFSDefault(). Until all functions are patched to use unicode strings, Python3 will not be fully unicode compliant *on Windows*. The problem is specific to Windows, because Python uses mbcs codec which doesn't support surrogateescape error handler.
I think that this patch is already huge and complex, and it will be difficult to fix all issues at the same time. This patch does improve the situation: with the patch, Python is fully unicode compliant (except on Windows), and it fixes at least one issue on Windows (bpo-8988, it now uses the right encoding).
vstinner commentedon Aug 7, 2010
The patch is too huge to be commited at once. I will split it again into smaller parts.
First related commit: r83778 fixes tests for not encodable filenames.
vstinner commentedon Aug 7, 2010
r83779 creates run_command(), it's just a refactorization.
vstinner commentedon Aug 8, 2010
_Py_wchar2char.patch: create _Py_wchar2char() private function, and _wstat() and _wfopen() use it. _Py_wchar2char() function has been improved since the previous version posted to Rietveld: it now computes the exact length of the output buffer, instead of using wcslen(text)*10+1.
Alone, this patch isn't really useful, but it prepares the code for next patches.
vstinner commentedon Aug 8, 2010
r83783 creates run_file() subfunction.
vstinner commentedon Aug 8, 2010
pyerr_warnformat.patch: create PyErr_WarnFormat() function, and use it in PyType_Ready() and PyUnicode_AsEncodedString(). The patch fixes also setup_context(): work on the unicode filename, not the encoded (bytes) filename. It does fix a bug because len is a number of characters, not a number of bytes: the number of bytes is bigger than the number of characters if the filename contains a non-ASCII character.
Advantages of PyErr_WarnFormat() over PyOS_snprintf() + PyErr_WarnEx():
Differences with Rietveld's version: rename PyErr_WarnUnicode() to warn_unicode() (it's now a static function) and document PyErr_WarnFormat().
vstinner commentedon Aug 8, 2010
nullimporter_unicode.patch: patch NullImporter_init():
pitrou commentedon Aug 8, 2010
It looks like you are a fixing a bug in setup_context() at the same time as you introduce PyErr_WarnFormat(). Both changes should probably go in separately.
The PyErr_WarnFormat() doc needs a "versionadded" tag.
32 remaining items
vstinner commentedon Aug 16, 2010
Py_UNICODE_strncmp.patch was wrong for n=0. New version based on libiberty/strncmp.c source code.
vstinner commentedon Aug 16, 2010
Py_UNICODE_strncmp-2.patch commited as r84111.
vstinner commentedon Aug 16, 2010
r84120: get_data() function of zipimport uses an unicode path.
vstinner commentedon Aug 17, 2010
r84121: repr() method zipimporter object uses unicode.
vstinner commentedon Aug 17, 2010
r84122 saves/restores the exception around "filename = _PyUnicode_AsString(co->co_filename);" because it raises an unicode error on unencodable filename.
vstinner commentedon Aug 17, 2010
r84168 creates PyModule_GetFilenameObject().
I created a separated issue for the patch reencoding all filenames when setting the filesystem encoding: bpo-9630.
vstinner commentedon Aug 24, 2010
See also bpo-1552880.
kristjanvalur commentedon Aug 24, 2010
Yes. in bpo-1552880 I tried to make as minimal a change as possible. This particular patch is still in use in EVE Online, which is installed in various strange and exotic paths in the orient..
The trick I employed there was to encode everything to utf-8 at the earliest oppertunity (current working directory, any unicode members in sys.path, etc.) and let the import.c machinery crunch that utf-8 code. This works because path separators and other such stuff doesn't change under the utf-8 encoding. As a final step, the utf8 encoded working string is converted back to unicode and native unicode API calls (on windows) are used to stat() and open() files.
A similar trick could be used on unix by converting from utf-8 to whatever native encoding the stat() and open() calls expect.
My patch never got accepted because I didn't have the time to put in the extra effort to make it cross platform.
vstinner commentedon Aug 25, 2010
oops, it's r84013 (not r84012)
vstinner commentedon Aug 29, 2010
Py_UNICODE_strcat.patch: create Py_UNICODE_strcat() function.
Py_UNICODE_strdup.patch: create Py_UNICODE_strdup() function.
vstinner commentedon Sep 1, 2010
r84429 creates Py_UNICODE_strcat() (change with the patch: return the right value).
r84430 creates PyUnicode_strdup() (change with the patch: rename the function from Py_UNICODE_strdup() to PyUnicode_strdup() and mangle the function name).
vstinner commentedon Sep 29, 2010
r85115 closes bpo-9630: an important patch for bpo-9425, redecode all filenames when setting the filesystem encoding.
Next tasks (maybe not in this order):
vstinner commentedon Oct 19, 2010
Starting at r85691, the full test suite of Python 3.2 pass with ASCII, ISO-8859-1 and UTF-8 locale encodings in a non-ascii directory. The work on this issue is done.