Repository navigation
locale.py resetlocale throws exception on Windows (getdefaultlocale returns value not usable in setlocale) #54675
Description
| BPO | 10466 |
|---|---|
| Nosy | @malemburg, @loewis, @pfmoore, @tjguk, @ned-deily, @bitdancer, @briancurtin, @zware, @zooba, @vidartf |
| Superseder |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-09-11.09:49:22.616>
created_at = <Date 2010-11-20.11:36:05.386>
labels = ['3.7', 'type-bug', 'library', 'OS-windows']
title = 'locale.py resetlocale throws exception on Windows (getdefaultlocale returns value not usable in setlocale)'
updated_at = <Date 2019-09-11.09:49:22.615>
user = 'https://bugs.python.org/skoczian'bugs.python.org fields:
activity = <Date 2019-09-11.09:49:22.615>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-11.09:49:22.616>
closer = 'steve.dower'
components = ['Library (Lib)', 'Windows']
creation = <Date 2010-11-20.11:36:05.386>
creator = 'skoczian'
dependencies = []
files = []
hgrepos = []
issue_num = 10466
keywords = []
message_count = 17.0
messages = ['121616', '121656', '121907', '121961', '122001', '122065', '122071', '122090', '122119', '122127', '122216', '122217', '136036', '220349', '220350', '350303', '351786']
nosy_count = 14.0
nosy_names = ['lemburg', 'loewis', 'paul.moore', 'tim.golden', 'ned.deily', 'r.david.murray', 'brian.curtin', 'iki', 'guy.linton', 'skoczian', 'Gabi.Davar', 'zach.ware', 'steve.dower', 'vidartf']
pr_nums = []
priority = 'low'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '37945'
type = 'behavior'
url = 'https://bugs.python.org/issue10466'
versions = ['Python 2.7', 'Python 3.7']
Activity
skoczian commentedon Nov 20, 2010
Running locale.py as a module on Windows, using the Python command window, produces this output:
C:\Python31\Lib>locale.py
Locale aliasing:
Locale defaults as determined by getdefaultlocale():
------------------------------------------------------------------------
Language: de_DE
Encoding: cp1252
Locale settings on startup:
------------------------------------------------------------------------
LC_NUMERIC ...
Language: (undefined)
Encoding: (undefined)
LC_MONETARY ...
Language: (undefined)
Encoding: (undefined)
LC_COLLATE ...
Language: (undefined)
Encoding: (undefined)
LC_CTYPE ...
Language: (undefined)
Encoding: (undefined)
LC_TIME ...
Language: (undefined)
Encoding: (undefined)
Locale settings after calling resetlocale():
------------------------------------------------------------------------
Tried with 2.7 on Windows XP, 32bit, 3.1.2 and 3.2a4 on Windows 7, 64bit. System character set in all cases cp1252 as usual for a German windows installation.
With 2.6, 2.6.5, 3.1.2 on Linux with UTF-8 system character set: no exception, expected output.
bitdancer commentedon Nov 20, 2010
unsupported locale setting is a message that comes from the C runtime, IIUC. Does it work on windows with 2.6?
skoczian commentedon Nov 21, 2010
-----Ursprüngliche Nachricht-----
Von: "R. David Murray" <report@bugs.python.org>
Gesendet: 20.11.2010 15:37:28
An: nulla.epistola@web.de
Betreff: [bpo-10466] locale.py throws exception on Windows / Non-UTF8 system
No, it doesn't. I installed 2.6.6, and I found an old 2.3 (installed by some application). Both of them throw the same exception if locale is run as a script.
GRATIS! Movie-FLAT mit über 300 Videos.
Jetzt freischalten unter http://movieflat.web.de
bitdancer commentedon Nov 21, 2010
See also bpo-1699853.
The problem here is that resetlocale calls getdefaultlocale, and getdefaultlocale returns something that setlocale cannot consume on Windows. For example, on my US windows system, getlocale returns ('English_United States', '1252'), while getdefaultlocale returns ('en_US', 'cp1252'). The latter results in the 'unsupported locale settting' error message when passed to setlocale.
Since getdefaultlocale contains special-case code for the win32 platform, I wonder if this is a bug in getdefaultlocale. I suspect it is operating as designed, but is that operation useful if the result isn't recognized by the platform in question? Or if, as it appears from looking at the Python code (but I haven't looked at the C code) the data is actually coming from the C runtime, perhaps the bug is that setlocale doesn't accept the output of getdefaultlocale on windows?
Hopefully Mark Andre will have an answer, but I've also added our windows experts to nosy to see if they have any insights.
Given the long standing nature of this issue, it seems that the priority isn't very high, especially since the bug is revealed not by user code but by a diagnostic routine in the stdlib.
[-]locale.py throws exception on Windows / Non-UTF8 system[/-][+]locale.py resetlocale throws exception on Windows[/+]skoczian commentedon Nov 21, 2010
Am 21.11.2010 20:04, schrieb R. David Murray:
It's true that the bug is quite old, but first it crashed my
application, then I found that locale.py doesn't run as a script. I put
that into the tracker simply because it's so easy to reproduce.
But this is almost as simple (and it's the same bug, isn't it?):
Python 3.1.2 (r312:79149, Mar 20 2010, 22:55:39) [MSC v.1500 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import calendar >>> kal = calendar.localeHTMLCalendar() Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'module' object has no attribute 'localeHTMLCalendar' >>> kal = calendar.LocaleHTMLCalendar() >>> kal_aus = kal.formatyearpage(2011) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Python31\lib\calendar.py", line 478, in formatyearpage a(self.formatyear(theyear, width)) File "C:\Python31\lib\calendar.py", line 454, in formatyear a(self.formatmonth(theyear, m, withyear=False)) File "C:\Python31\lib\calendar.py", line 427, in formatmonth a(self.formatmonthname(theyear, themonth, withyear=withyear)) File "C:\Python31\lib\calendar.py", line 546, in formatmonthname with different_locale(self.locale): File "C:\Python31\lib\calendar.py", line 489, in __enter__ self.oldlocale = _locale.setlocale(_locale.LC_TIME, self.locale) File "C:\Python31\lib\locale.py", line 527, in setlocale return _setlocale(category, locale) locale.Error: unsupported locale setting >>>This would be quite unfunny if I couldn't run my script on Linux instead.
Sibylle
bitdancer commentedon Nov 22, 2010
It is unfunny that your program fails on Windows.
Yes it is the same bug, since calendar calls getdefaultlocale.
bpo-1080864 makes very interesting reading in this context. Clearly Martin is right, yet the OP seems to find the current behavior of getdefaultencoding (returning the more standard locale name rather than Windows unique values) useful.
It is an interesting question what can be done about this. Clearly resetlocale is non-functional on Windows, and getdefaultlocale is of questionable utility. At the very least these limitations need to be documented.
Your application bug, however, can probably be addressed by fixing calendar to not use getdefaultencoding. We should open a new bug for that, though.
[-]locale.py resetlocale throws exception on Windows[/-][+]locale.py resetlocale throws exception on Windows (getdefaultlocale returns value not usable in setlocale)[/+]bitdancer commentedon Nov 22, 2010
I've opened bpo-10498 for the calendar bug.
ned-deily commentedon Nov 22, 2010
FYI, calendar.localeHTMLCalendar() fails on OS X in exactly the same way for exactly the same reason. See, for instance, bpo-10090.
malemburg commentedon Nov 22, 2010
I think that's a bug in the resetlocale() API.
The correct way to reset the locale setting to defaults, it to use
The other issues here is that getlocale() appears to return non-ISO language codes on Windows. If that's indeed the case, then we would need to add mappings of the Windows codes to the ISO ones and use a reverse mappings to make setlocale() work with the ISO codes.
Perhaps it's easier to just update the mapping used by getdefaultencoding() to return the non-ISO codes used by Windows and then update the documentation to say that on Windows, non-ISO codes are returned.
BTW: I wonder why this hasn't popped up earlier.
bitdancer commentedon Nov 22, 2010
I had a report from a user on IRC during the bug weekend that they could not reproduce the failure on windows. So it may be dependent on the windows version. That doesn't answer your question of why it hasn't come up before, though, since my tests were done on XP.
malemburg commentedon Nov 23, 2010
R. David Murray wrote:
Some research shows that the MS VCRT uses non-ISO locale names
for setlocale():
http://msdn.microsoft.com/en-us/library/x99tb11d.aspx
http://msdn.microsoft.com/en-us/library/hzz3tw78.aspx
http://msdn.microsoft.com/en-us/library/39cwe7zf.aspx
http://msdn.microsoft.com/en-us/library/cdax410z.aspx
and it doesn't support the ISO style locale namings, even though
the setlocale() page says "The set of available languages, country/region codes, and code pages
includes all those supported by the Win32 NLS API" and the Win32
API does support the ISO names:
http://msdn.microsoft.com/en-us/library/dd373814(VS.85).aspx
I'll have to check whether Vista also shows this behavior. On
Win XP the setlocale() API doesn't accept ISO language names;
I can confirm that. It only accepts the fully written out
form described on the above pages.
malemburg commentedon Nov 23, 2010
Marc-Andre Lemburg wrote:
Confirmed on Vista as well.
I think the only choice we have is to add a new alias dictionary
mapping the ISO language names to the Windows ones.
guylinton commentedon May 15, 2011
Surely the user of getlocale as the right to expect that the same thing (i.e. ISO language codes) would be returned on all platforms. I am looking at some code that provides special purpose date handling routines. The appropriate routine is selected by a language code (for the locale category LC_TIME as it happens) like fr_FR. In order to get this to work on different platforms, you have to test whether it is windows, and if so use getdefaultlocale, while for other platforms, you use getlocale in order to return similar language codes. This code would actually be wrong if the application were to change the locale inside the application, because on other platforms the change would have an effect, while on windows, it would be ignored! [I am running on Mac OS X as it happens, not Linux or Windows].
So, don’t say that getdefaultloale is not useful, it is the one that is needed on Windows!
Note this is not a problem with the encoding – some of the discussion and many of the related bugs are concerned with the encoding.
Also note Marc-Andre pointing out in http://mail.python.org/pipermail/python-bugs-list/2004-December/026667.html that “getdefaultlocale() mimics the lookup mechanism of setlocale(LC_ALL, "")”. I recognise that he is talking about the lookup mechanism rather than the results, but it seems to suggest that the results might be similar (if you do the same lookup you would get the same result). But “getdefaultlocale returns something that setlocale cannot consume on Windows”!
I recognise that it would be difficult to decide to code locale on python so that windows returns something that is not the same as the native call, but it does seem to me that python locale should translate the Windows language codes so that they are the same as are returned on all the other platforms (i.e. ISO codes etc).
bitdancer commentedon Jun 12, 2014
See bpo-21731 for considering putting a workaround for this into the calendar module (noted here because of msg122065).
bitdancer commentedon Jun 12, 2014
Oh, I see I'd already previously opened bpo-10498 for that.
vidartf commentedon Aug 23, 2019
Note that this is still an issue on Windows:
>>> import locale >>> locale.getdefaultlocale() ('en_US', 'cp1252') >>> locale.setlocale(locale.getdefaultlocale()) Error: unsupported locale settingzooba commentedon Sep 11, 2019
bpo-37945 is a duplicate and has more up-to-date information on it, so I'm going to close this one even though it's older.