Repository navigation
Full unicode import system #47330
Copy link
Copy link
Closed
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
| BPO | 3080 |
|---|---|
| Nosy | @brettcannon, @birkenfeld, @terryjreedy, @amauryfa, @ncoghlan, @abalkin, @pitrou, @vstinner, @benjaminp, @merwok, @bitdancer, @asvetlov |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/vstinner'
closed_at = <Date 2011-03-21.00:01:35.123>
created_at = <Date 2008-06-11.18:24:56.218>
labels = ['interpreter-core', 'type-bug']
title = 'Full unicode import system'
updated_at = <Date 2011-03-23.15:58:18.116>
user = 'https://github.com/amauryfa'bugs.python.org fields:
activity = <Date 2011-03-23.15:58:18.116>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2011-03-21.00:01:35.123>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2008-06-11.18:24:56.218>
creator = 'amaury.forgeotdarc'
dependencies = []
files = ['20477', '21296', '21305']
hgrepos = []
issue_num = 3080
keywords = ['patch']
message_count = 60.0
messages = ['68005', '68015', '109844', '112028', '119107', '123963', '123993', '124756', '125752', '126514', '126515', '126516', '126606', '126608', '126612', '126613', '126672', '126673', '126676', '126678', '126680', '126681', '126695', '126705', '126706', '126708', '126752', '126755', '126756', '126760', '127591', '127674', '129141', '129143', '129185', '129196', '130050', '130473', '130492', '130507', '130645', '130935', '131457', '131464', '131472', '131473', '131474', '131483', '131484', '131516', '131547', '131571', '131572', '131575', '131606', '131612', '131625', '131664', '131711', '131890']
nosy_count = 14.0
nosy_names = ['brett.cannon', 'georg.brandl', 'terry.reedy', 'amaury.forgeotdarc', 'ncoghlan', 'belopolsky', 'pitrou', 'vstinner', 'benjamin.peterson', 'eric.araujo', 'Arfrever', 'r.david.murray', 'asvetlov', 'python-dev']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue3080'
versions = ['Python 3.3']Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Activity
ncoghlan commentedon Jan 20, 2011
Victor, could you please create a Reitveld review for this? The auto-review creator can't cope with the Git diffs.
vstinner commentedon Jan 20, 2011
Yes, but not yet. I have first to cleanup the patch.
ncoghlan commentedon Jan 20, 2011
OK - I'll wait until that is ready before digging into this.
vstinner commentedon Jan 20, 2011
See also bpo-8754: repr() is better than str() for other reasons, eg. to see a space at the end of a module name (import('space ')) thanks to the quotes.
vstinner commentedon Jan 21, 2011
Version 4 of the patch.
vstinner commentedon Jan 21, 2011
Same patch (version 4) generated by svn.
vstinner commentedon Jan 21, 2011
You can review the patch with Rietveld:
http://codereview.appspot.com/3972045
vstinner commentedon Jan 21, 2011
Oops, there is a dummy typo in imp_init_builtin() that makes test_importlib to crash (which proves that importlib has a good coverage :-)): replace "s:" by "U:" in if (!PyArg_ParseTuple(args, "s:init_builtin", &name)).
vstinner commentedon Jan 21, 2011
test_reprlib fails on Windows, because '\' in quoted '\\' in the filename on repr(module). Workaround:
It is maybe not a good idea to use %R to format the filename in module.__repr__().
vstinner commentedon Jan 21, 2011
test_runpy fails on Windows on make_legacy_pyc() (of test.support), I don't know why.
ncoghlan commentedon Jan 21, 2011
After applying the patch, doing a make clean and rebuild, I found that test_importlib fails with a segmentation fault, but the default test suite otherwise runs without error (that's on Linux with a UTF-8 filesystem, though).
I'll see how a -uall run fares.
ncoghlan commentedon Jan 21, 2011
As for the more limited run, I get a clean run with -uall except for the segfault in test_importlib.
I'll switch to a pydebug build and see how a verbose run of that test fares.
ncoghlan commentedon Jan 21, 2011
I haven't investigated in detail yet, but this is the final line showing the failing test:
test_module (importlib.test.builtin.test_loader.LoaderTests) ... Segmentation fault
vstinner commentedon Jan 21, 2011
Yes, as reported in my previous comment :-) Let's update the patch for practical reasons. But I don't want to touch http://codereview.appspot.com/1874048 (based on patch version 4).
ncoghlan commentedon Jan 21, 2011
Oops, missed that post - that was indeed the problem. With that fixed, tests are all good on this system. I'll give the patch a look anyway, but I'm going to have trouble diagnosing things that don't fail on my development machine.
As far as the test_reprlib failure goes, I seem to recall addressing a similar problem elsewhere in the standard lib by replace a "%r" code with "'%s'" to get the single quotes without the backslash escaping. A similar change should probably do the trick here.
vstinner commentedon Jan 21, 2011
On Windows, try any character not encodable into your ANSI code page (eg. Ł with cp1252) in the module path and non-ASCII characters in the module name.
On Mac OS X, sorry, it already works.
On other OSes, set the locale to something else than UTF-8 (and than ASCII because ASCII is not very interesting) and try non-ASCII module names. The patch includes bpo-3080.py: set the locale to fr_FR.iso88591 to have ISO-8859-1 as locale encoding and try to load a module called 'bpo-3080\xE4'. U+00E4 is encoded to b'\xE4' in ISO-8859-1 and b'\xC3\xA4' to UTF-8.
vstinner commentedon Jan 21, 2011
I tried bpo-3080-5.patch. The whole test suite pass on Windows. It pass also on Linux with "-Wd -Werror -R 3:3:" (except bpo-10971 which is unrelated to this issue).
I should maybe add some unit tests for non-ASCII module paths and non-ASCII module names :-)
ncoghlan commentedon Jan 21, 2011
On Sat, Jan 22, 2011 at 3:08 AM, STINNER Victor <report@bugs.python.org> wrote:
Indeed. There are a few tests in test_runpy that could be adapted to
that task fairly easily (it creates the test packages at run time in a
temporary directory, so copying that to make a non-ASCII path and
module name test should be too difficult).
vstinner commentedon Jan 31, 2011
As explained in issue bpo-10828: Python 3.2 doesn't support non-ASCII module names on Windows because module names are encoded to UTF-8 instead of the filesystem encoding (the ANSI code page).
vstinner commentedon Feb 1, 2011
See also bpo-6011.
vstinner commentedon Feb 22, 2011
I started to commit some parts of the huge patch:
r88515: Mark PyWin_FindRegisteredModule() as private
r88516: Remove unused argument of _PyImport_GetDynLoadFunc()
r88517 (3.3), r88518 (3.2): document encoding used by import functions
vstinner commentedon Feb 23, 2011
r88519: Mark _PyImport_FindBuiltin() argument as constant
r88520: Add PyModule_GetNameObject()
pitrou commentedon Feb 23, 2011
This new failure is perhaps related:
http://www.python.org/dev/buildbot/all/builders/AMD64%20Windows%20Server%202008%203.x/builds/572/steps/test/logs/stdio
======================================================================
FAIL: test_module (test.test_reprlib.LongReprTest)
----------------------------------------------------------------------
vstinner commentedon Feb 23, 2011
Ah yes, yesterday, I tried to remember which test was impacted by the module change, but all tests passed on Linux. Anyway, it's now fixed by r88533.
vstinner commentedon Mar 4, 2011
r88746: Add PyModule_NewObject() function
r88747: Add PyImport_AddModuleObject() and PyImport_ExecCodeModuleObject()
vstinner commentedon Mar 9, 2011
I created the features/unicode_import repository with a "unicode_import" branch:
http://hg.python.org/features/unicode_import/
It's my huge patch splitted into small and atomic commits.
amauryfa commentedon Mar 10, 2011
Nice work! Is there a specific place for comments? Here are some of them already:
look like:
for example, in case the pathname cannot be encoded to mbcs?
vstinner commentedon Mar 10, 2011
Yes, but my work is not done. I still have parts to commit.
Oh. This one is not easy because this function has many implementations and all implementations have the same prototype. I will maybe fix it later.
vstinner commentedon Mar 11, 2011
See also bpo-9319: when this issue will be fixed, it will be easier to fix bpo-9319.
vstinner commentedon Mar 15, 2011
I finished to split the huge patch into smaller commits. You can now test the unicode_import Mercurial branch. Especially, it should be tested on Windows.
I don't know if I should merge the branch as an unique commit or as multiple commits. Some of them can be simply be merged.
You can try bpo-3080.py (file attached to this issue, extracted from the patch): a short script testing this issue.
--
The parser and _PyImport_GetDynLoadFunc() (on Windows) do still store the filename as byte strings, and so I don't think that Python is ready to use full Unicode range for filenames on Windows. But at least, it should now support non-ASCII module names and paths which are encodable to the ANSI code page.
Issue bpo-10785 should improve the situation at least for the parser.
But for _PyImport_GetDynLoadFunc(), I don't know if there is a Unicode version of LoadLibraryEx().
--
Implemented in f286d3b514e0.
Done in 76907d413b99
vstinner commentedon Mar 19, 2011
I reverted this change in my Mercurial branch (unicode_import).
done
done
done
done: find_module_path_list() and find_module_path()
vstinner commentedon Mar 19, 2011
Gotcha: I replaced mkdir() by CreateDirectoryW(), but the "directory already exists" error was not ignored. Fixed by 2debe178697b.
python-dev commentedon Mar 20, 2011
New changeset 6c80ac44ae9c by Victor Stinner in branch 'default':
Issue bpo-3080: zipimport has a full unicode suppport
http://hg.python.org/cpython/rev/6c80ac44ae9c
New changeset b50a0d44545a by Victor Stinner in branch 'default':
Issue bpo-3080: PyImport_Cleanup() uses Unicode
http://hg.python.org/cpython/rev/b50a0d44545a
New changeset e7c1019b27b9 by Victor Stinner in branch 'default':
Issue bpo-3080: Add PyImport_ImportFrozenModuleObject()
http://hg.python.org/cpython/rev/e7c1019b27b9
New changeset 2425717c6430 by Victor Stinner in branch 'default':
Issue bpo-3080: Import builtins using Unicode strings
http://hg.python.org/cpython/rev/2425717c6430
New changeset ced52fcd95f6 by Victor Stinner in branch 'default':
Issue bpo-3080: Use PyUnicode_InternFromString() for builtins
http://hg.python.org/cpython/rev/ced52fcd95f6
New changeset e63a583ec689 by Victor Stinner in branch 'default':
Issue bpo-3080: Document the name attribute of the _inittab structure
http://hg.python.org/cpython/rev/e63a583ec689
New changeset bab42673674a by Victor Stinner in branch 'default':
Issue bpo-3080: _PyWin_FindRegisteredModule() returns the path as Unicode
http://hg.python.org/cpython/rev/bab42673674a
New changeset ef2b6305d395 by Victor Stinner in branch 'default':
Issue bpo-3080: _PyImport_LoadDynamicModule() uses Unicode for name and path
http://hg.python.org/cpython/rev/ef2b6305d395
New changeset d52f471fbbeb by Victor Stinner in branch 'default':
Issue bpo-3080: find_module() initialize buf and *p_fp
http://hg.python.org/cpython/rev/d52f471fbbeb
New changeset bdf5820f5a39 by Victor Stinner in branch 'default':
Issue bpo-3080: Remove useless name buffer from find_module()
http://hg.python.org/cpython/rev/bdf5820f5a39
New changeset a4d797b9ff63 by Victor Stinner in branch 'default':
Issue bpo-3080: Create find_module_path_list() subfunction
http://hg.python.org/cpython/rev/a4d797b9ff63
New changeset 09aaac73d9cf by Victor Stinner in branch 'default':
Issue bpo-3080: Create find_module_path() subfunction
http://hg.python.org/cpython/rev/09aaac73d9cf
New changeset f6507eb8e689 by Victor Stinner in branch 'default':
Issue bpo-3080: get_sourcefile(), make_source_pathname(), load_package()
http://hg.python.org/cpython/rev/f6507eb8e689
New changeset d24decc8c97e by Victor Stinner in branch 'default':
Issue bpo-3080: Use Unicode to import source and compiled modules
http://hg.python.org/cpython/rev/d24decc8c97e
New changeset 64c21f364519 by Victor Stinner in branch 'default':
Issue bpo-3080: load_module() expects name and path as Unicode
http://hg.python.org/cpython/rev/64c21f364519
New changeset e55e7f197649 by Victor Stinner in branch 'default':
Issue bpo-3080: PyImport_ImportModuleNoBlock() uses Unicode
http://hg.python.org/cpython/rev/e55e7f197649
New changeset 7c67aa3ab531 by Victor Stinner in branch 'default':
Issue bpo-3080: Use Unicode for the "The Magnum Opus of dotted-name import"
http://hg.python.org/cpython/rev/7c67aa3ab531
New changeset 23fe237afa81 by Victor Stinner in branch 'default':
Issue bpo-3080: Use %R to format module name in error messages
http://hg.python.org/cpython/rev/23fe237afa81
New changeset 2ee0ab9d2e8a by Victor Stinner in branch 'default':
Issue bpo-3080: Reindent and simplify import_submodule()
http://hg.python.org/cpython/rev/2ee0ab9d2e8a
New changeset 340f76a6a792 by Victor Stinner in branch 'default':
Issue bpo-3080: Drop OS/2 support for the import machinery
http://hg.python.org/cpython/rev/340f76a6a792
New changeset 156818529636 by Victor Stinner in branch 'default':
Issue bpo-3080: find_module() expects module fullname and subname as Unicode
http://hg.python.org/cpython/rev/156818529636
New changeset fe1d421ca3fa by Victor Stinner in branch 'default':
Issue bpo-3080: Rename some path variables to path_list
http://hg.python.org/cpython/rev/fe1d421ca3fa
New changeset c1a5a7dca1ec by Victor Stinner in branch 'default':
Issue bpo-3080: find_module() sets an empty path for builtin and frozen modules
http://hg.python.org/cpython/rev/c1a5a7dca1ec
New changeset c4ccf02456d6 by Victor Stinner in branch 'default':
Issue bpo-3080: Refactor find_module_path(), use return instead of break
http://hg.python.org/cpython/rev/c4ccf02456d6
New changeset 298a70b27497 by Victor Stinner in branch 'default':
Issue bpo-3080: find_init_module() expects Unicode
http://hg.python.org/cpython/rev/298a70b27497
New changeset 066b399a8477 by Victor Stinner in branch 'default':
Issue bpo-3080: case_ok() expects Unicode strings
http://hg.python.org/cpython/rev/066b399a8477
New changeset 9aec6f0e4076 by Victor Stinner in branch 'default':
Issue bpo-3080: find_module() returns the path as Unicode
http://hg.python.org/cpython/rev/9aec6f0e4076
New changeset c17bc2026145 by Victor Stinner in branch 'default':
Issue bpo-3080: imp.new_module() uses Unicode
http://hg.python.org/cpython/rev/c17bc2026145
New changeset c4361bab6914 by Victor Stinner in branch 'default':
Issue bpo-3080: Use repr() to format the module name on error
http://hg.python.org/cpython/rev/c4361bab6914
New changeset 80f4bd647695 by Victor Stinner in branch 'default':
Issue bpo-3080: Add PyImport_ImportModuleLevelObject() function
http://hg.python.org/cpython/rev/80f4bd647695
New changeset cc7c0f6f60bf by Victor Stinner in branch 'default':
Issue bpo-3080: skip test_bdist_rpm if sys.executable is not encodable to UTF-8
http://hg.python.org/cpython/rev/cc7c0f6f60bf
python-dev commentedon Mar 20, 2011
New changeset f8d6f6797909 by Victor Stinner in branch 'default':
Issue bpo-3080: Fix case_ok() using case_bytes()
http://hg.python.org/cpython/rev/f8d6f6797909
python-dev commentedon Mar 20, 2011
New changeset dc38c4d65cd9 by Victor Stinner in branch 'default':
Issue bpo-3080: Fix call to case_ok() in find_init_module()
http://hg.python.org/cpython/rev/dc38c4d65cd9
vstinner commentedon Mar 20, 2011
http://www.python.org/dev/buildbot/all/builders/PPC%20Tiger%203.x/builds/1599/steps/test/logs/stdio
======================================================================
ERROR: testImpWrapper (test.test_importhooks.ImportHooksTestCase)
----------------------------------------------------------------------
vstinner commentedon Mar 20, 2011
The problem is that imp.find_module() now returns None as the filename, but imp.load_module() doesn't support None.
merwok commentedon Mar 20, 2011
Attached patch fixes a typo in Doc/c-api/import.rst. You can merge it in your next commit.
python-dev commentedon Mar 20, 2011
New changeset 7f4a4e393058 by Victor Stinner in branch 'default':
Issue bpo-3080: imp.load_module() accepts None for the module path
http://hg.python.org/cpython/rev/7f4a4e393058
vstinner commentedon Mar 20, 2011
Ok. Python 3.3 does now support non-ASCII characters in module paths and names on Windows, but only characters encodable to the ANSI code page. To support the full Unicode range, we should remove all calls to PyUnicode_EncodeFSDefault() on Windows:
a) parse_source_module() has to encode the filename because the parser has no function expecting a filename as a Python object. It uses currently PyParser_ASTFromFile().
b) write_compiled_module() encodes the filename to call open_exclusive(). I don't know how to implement open_exclusive() for Windows using Unicode filename: open() expects the filename as a byte string. Can we use _Py_fopen() (_wfopen)? Or do you need the O_EXCL flag?
c) _PyImport_LoadDynamicModule() encodes the filename for _PyImport_GetDynLoadFunc(). The prototype should be changed, but only on Windows, to accept a filename as a Unicode string.
Issue bpo-10785 is the right fix to (a). When bpo-10785 will be fixed, it will be easier to fix bpo-9319 crash.
vstinner commentedon Mar 20, 2011
Hum, the difficult part is to use Unicode in _PyImport_GetDynLoadFunc() for:
There is a LoadLibraryW() function, but it doesn't have a flag argument. And I suppose that the LOAD_WITH_ALTERED_SEARCH_PATH option is important.
vstinner commentedon Mar 21, 2011
Ok, I think that the most important part is now implemented in Python 3.3: use Unicode for module names and paths in the import machinery. Remaing parts are specific to Windows, and so I opened a new issue: bpo-11619. Let's close this 3 years old issue.
python-dev commentedon Mar 21, 2011
New changeset ee4e780a6b7a by Éric Araujo in branch 'default':
Fix a typo (see bpo-3080)
http://hg.python.org/cpython/rev/ee4e780a6b7a
asvetlov commentedon Mar 21, 2011
As I see Victor has dropped OS/2 support from Python/import.c
Perhaps file Python/dynload_os2.c should be removed also.
Not sure about other dynload_* files.
vstinner commentedon Mar 21, 2011
340f76a6a792 just removes few lines in import.c: they can easily be rewritten. And this commit doesn't drop completly the support of OS/2 from the import machinery, as you wrote: dynload_os2.c still exists.
If we drop completly the support of OS/2, it should be done completly using a PEP (I don't remember its number), and it should be discussed. At least with Andrew I MacIntyre :-)
asvetlov commentedon Mar 21, 2011
Understood. Sorry.
I thought Python support only Windows and posix (Linux, BSD, MacOSX etc) systems now, all other OSes are not maintained.
Anyway please don't care about that.
python-dev commentedon Mar 22, 2011
New changeset 15f9eca5e956 by Victor Stinner in branch 'default':
Issue bpo-3080: On DJGPP, case_bytes() returns -1 to signal an error if the file
http://hg.python.org/cpython/rev/15f9eca5e956
vstinner commentedon Mar 23, 2011
test the fixed nosy list