Repository navigation
Python 3 doesn't support non-ASCII module names with a locale encoding different than UTF-8 #55037
Description
| BPO | 10828 |
|---|---|
| Nosy | @terryjreedy, @vstinner, @bitdancer |
| Superseder |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2011-03-21.00:56:42.027>
created_at = <Date 2011-01-04.19:44:14.630>
labels = ['type-bug']
title = "Python 3 doesn't support non-ASCII module names with a locale encoding different than UTF-8"
updated_at = <Date 2011-03-21.00:56:42.026>
user = 'https://bugs.python.org/ingemar'bugs.python.org fields:
activity = <Date 2011-03-21.00:56:42.026>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2011-03-21.00:56:42.027>
closer = 'terry.reedy'
components = []
creation = <Date 2011-01-04.19:44:14.630>
creator = 'ingemar'
dependencies = []
files = []
hgrepos = []
issue_num = 10828
keywords = []
message_count = 15.0
messages = ['125360', '125366', '125381', '125408', '125739', '125745', '125753', '125754', '125786', '125787', '125795', '125819', '125822', '131577', '131588']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'vstinner', 'r.david.murray', 'ingemar']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = None
status = 'closed'
superseder = '11619'
type = 'behavior'
url = 'https://bugs.python.org/issue10828'
versions = ['Python 3.1', 'Python 3.2']
Activity
ingemar commentedon Jan 4, 2011
I have a set of programs written for Python3.1 and running well on Kubuntu. The source files are located on a Samba server on a Kubuntu box. Several of the programs contain Python/PyQt code to start other programs in the set ( QtCore.QProcess().startDetached(kommando) )
I have had no problems using non-ascii filenames in the Linux environment.
When I tried to check the programs in a MS Windows environment (Win2K with Python 3.1.2 in a VirtualBox in a Kubuntu box) then Python complained:
ImportError: module xxx not found..
The ugly solution has been to refrain from the use of non-ascii characters in the names of files imported from. This involved the filename of the imported file and also one line of code changed in the importing file.
Example:
"from gui_jämföra import * " ---> "from gui_jamfora import gui_Jämföra"
Is there a beautiful solution that will permit me to use non-ascii utf-8 also in the file names of files imported from?
bitdancer commentedon Jan 4, 2011
Have you tried 3.2b2?
vstinner commentedon Jan 4, 2011
I think that this issue is a duplicate of bpo-8611 (and bpo-9425), it should be fixed in Python 3.2.
ingemar commentedon Jan 5, 2011
Have I tried 3.2b2?
No. I will have to wait for 3.2, or more exactly for a Windows installer for PyQt for 3.2 to become available.
Compiling that on Windows is beyond my resources and experience.
I will make a point to tell you then.
terryjreedy commentedon Jan 8, 2011
(Ingemar: one can easily test import statements without pyqt, let alone qt ;-)
With 3.2b2 on our Win7, 64 bit machine, files with a Japanese name run but apparently cannot be imported.
a.py: print('something')
^|.py: print('other') # ^| == imitation of katakana name
c.py: import a; import ^|
something
ImportError: No module named ^|
Tried in both japanese- and then ascii-named directories.
So I am not convinced that bpo-9425 is finished. What might I have misunderstood?
vstinner commentedon Jan 8, 2011
What is your ANSI code page? If it is not a japanese code page, it is the issue bpo-3080.
On Windows, bpo-8611 (and bpo-9425) permit to use non-ASCII characters in the module path... but only characters encodable to your ANSI code page.
terryjreedy commentedon Jan 8, 2011
ANSI code page? I have no idea how to find out and many would not even know what such a thing exists. It is an HP laptop sold in the US.
I think bugs in core syntax should have high priority. I appreciate your work toward fixing it.
ingemar commentedon Jan 8, 2011
Terry: Thanks for the hint
In a pure ascii path I created files very similar to yours with Swedish "ä" instead of your katakana character.
I also got the same result.
a.py:
print ('something')
ä.py:
print ('other')
c.py:
# -- coding: utf-8 --
import a
import ä
I ran the files with 3.2b2:
c:\Python32\python.exe a.py
something
c:\Python32\python.exe ä.py
other
Victor: How do I determine what code page my old w2k is using?.
Would that be 8859-1 or some older variant for western Europe or Sweden?
vstinner commentedon Jan 8, 2011
python.exe -c 'import locale; print("ANSI code page: {}".format(locale.getpreferredencoding()))'
If you would like to check if your path is encodable to your ANSI code page, try:
python.exe -c "import os; fn=os.fsencode('ä'); print(ascii(fn))"
If fsencode() raises an error, the filename is not encodable to your ANSI code page and you have to wait until bpo-3080 is fixed :-)
vstinner commentedon Jan 8, 2011
It took me 7 months to implement the first part (bpo-8611 and bpo-9425). I plan to do the second part (bpo-3080) in Python 3.3 (it's too late for Python 3.2, final is planned for February 5, 2011). I already have an huge patch somewhere (in a SVN branch, import_unicode), but I have to update the patch and split it into small and simple patches.
ingemar commentedon Jan 8, 2011
python.exe -c "import locale; print('ANSI code page: {}'.format(locale.getpreferredencoding()))"
ANSI code page: cp1252
python.exe -c "import os; fn=os.fsencode('ä'); print(ascii(fn))"
b'\xe4'
and no error raised
vstinner commentedon Jan 9, 2011
Hum, I ran your example with a debugger, and ok, I now remember the whole thing.
I fixed Python to support non-ASCII characters (... only non-ASCII characters encodable to the ANSI code page for Windows) in the *search path*, not in the module name.
The import machinery encodes each search path to the filesystem encoding, but it encodes the module name to UTF-8. Concatenate two byte strings encoded to different encodings doesn't work (it leads to mojibake).
To fix this problem, there are two solutions:
a) encode the module name to the fileystem encoding
b) manipulate paths as unicode strings; to access the filesystem: use the wide character (unicode) API of Windows and encode paths to the filesystem encoding on UNIX/BSD
It is easier to implement (a) than (b), but (a) only gives you the support of paths and module names encodable to the ANSI code page.
(b) gives you the full unicode support because it never *encodes* paths to the filesystem encoding, but it may *decodes* paths from the filesystem encoding. Encode a path raises a UnicodeEncodeError on the first character not encodable to the ANSI code page, whereas decode a path never fails (except if the user manually changed its code page to a rare ANSI code page like UTF-8).
I implemented (b) in my import_unicode SVN branch, but as I wrote, I still have some work to merge this branch into py3k, and anyway I will wait for Python 3.3.
ingemar commentedon Jan 9, 2011
Thanks Victor for the explanation.
Py3 is still far better than Py2, letting me use utf-8 as much as it does.
I will be able to live with this bug being known. I can understand though, that people in some places of the world may feel more concerned.
[-]Cannot use nonascii utf8 in names of files imported from[/-][+]Python 3 doesn't support non-ASCII module names with a locale encoding different than UTF-8[/+]vstinner commentedon Mar 21, 2011
I closed bpo-3080: Python 3.3 is now able to handle non-ASCII characters in module names and paths. But it is only able to handle non-ASCII characters encodable to the ANSI code page. To support all characters, I opened the issue bpo-11619 (see also bpo-10785).
terryjreedy commentedon Mar 21, 2011
As Victor noted, this issue is essentially a duplicate of bpo-3080 (and others) and now bpo-11619 and needs no independent action apart from the latter. Since the discussion with ingemar seems finished, I am now closing.