Repository navigation
localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding #76081
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2018-11-28.16:53:58.445>
created_at = <Date 2017-10-30.13:41:05.693>
labels = ['3.8', '3.7', 'tests']
title = 'localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding'
updated_at = <Date 2018-11-28.16:53:58.444>
user = 'https://github.com/stratakis'bugs.python.org fields:
activity = <Date 2018-11-28.16:53:58.444>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-28.16:53:58.445>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-10-30.13:41:05.693>
creator = 'cstratak'
dependencies = []
files = ['47246', '47377', '47386']
hgrepos = []
issue_num = 31900
keywords = ['patch']
message_count = 33.0
messages = ['305227', '305229', '305230', '305231', '305235', '305236', '305237', '307230', '308561', '309774', '309960', '309962', '309966', '309969', '309970', '309971', '309973', '309974', '309975', '309977', '309978', '309980', '309981', '309986', '309987', '309988', '309989', '309993', '310020', '310940', '327905', '330610', '330611']
nosy_count = 6.0
nosy_names = ['lemburg', 'vstinner', 'skrah', 'serhiy.storchaka', 'schwab', 'cstratak']
pr_nums = ['4174', '5191', '5192']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue31900'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']
Activity
stratakis commentedon Oct 30, 2017
Original bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1484497
It seems that on the development branch of Fedora, when we updated glibc from 2.26 to 2.26.90, test_float_with_comma started failing.
Details from the original bug report:
Under certain circumstances, when LC_NUMERIC is fr_FR.ISO8859-1 but LC_ALL is C.UTF-8, locale.localeconv() fails with
UnicodeDecodeError: 'locale' codec can't decode byte 0xa0 in position 0: Invalid or incomplete multibyte or wide character
Apparently, the thousands separator (or something else) in the lconv is "\xa0" (unbreakable space in fr_FR.ISO8859-1), and it's being decoded with UTF-8.
This is tripped by Python's test suite, namely test_float.GeneralFloatCases.test_float_with_comma
vstinner commentedon Oct 30, 2017
I can reproduce the bug with Python 3.6 on Fedora 26 and these locales:
Good: LC_NUMERIC = LC_CTYPE = LC_ALL = "es_MX.utf8"
haypo@selma$ env -i python3 -c 'import locale; locale.setlocale(locale.LC_ALL, "es_MX.utf8"); print(ascii(locale.localeconv()["thousands_sep"]))'
=> '\u2009'
Bug: LC_NUMERIC = "es_MX.utf8" but LC_CTYPE = LC_ALL = "fr_FR"
haypo@selma$ env -i python3 -c 'import locale; locale.setlocale(locale.LC_ALL, "fr_FR"); locale.setlocale(locale.LC_NUMERIC, "es_MX.utf8"); print(ascii(locale.localeconv()["thousands_sep"]))'
=> '\xe2\x80\x89'
[-]UnicodeDecodeError in localeconv() makes test_float fail with glibc 2.26.90[/-][+]localeconv() should decide numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding[/+]stratakis commentedon Oct 30, 2017
Tested the PR on a system with glibc 2.26.90 where the test was failing, and it successfully passed.
[-]localeconv() should decide numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding[/-][+]localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding[/+]serhiy-storchaka commentedon Oct 30, 2017
This is a duplicate of bpo-28604. See also bpo-25812.
skrah commentedon Oct 30, 2017
Same as bpo-7442, I think.
vstinner commentedon Oct 30, 2017
Oh wow, this bug is older than what I expected :-) I added support for non-ASCII thousands separator in 2012:
https://bugs.python.org/issue13706#msg151733
vstinner commentedon Oct 30, 2017
inconsistent_locale_encodings.py of closed issue bpo-7442 is interesting: I copy it here.
stratakis commentedon Nov 29, 2017
Pinging here. Is there some way I can help to move the issue forward?
vstinner commentedon Dec 18, 2017
Oh. Another Python function is impacted by the bug, str.format:
It should be '1\u2009000' ('1', '\u2009', '000').
vstinner commentedon Jan 10, 2018
I completed my change. It now fixes locale.localeconv(), str.format() for int, float, complex and decimal.Decimal:
vstinner@apu$ ./python lc_numeric.py
LC_CTYPE: ('fr_FR', 'ISO8859-1')
LC_NUMERIC: ('es_MX', 'UTF-8')
decimal_point: '.'
thousands_sep: '\u2009'
grouping: [3, 3, 0]
int.__format__: '1\u2009234'
float.__format__: '1\u2009234'
complex.__format__: '1\u2009234+0j'
Decimal.__format__: '1\u2009234'
10 remaining items
malemburg commentedon Jan 15, 2018
Indeed. The major problem with all libc locale functions is that they are not thread safe. The GIL does help a bit protecting against corrupted data, though.
vstinner commentedon Jan 15, 2018
Past 10 years, I repeated to every single user I met that "Python 3 is right, your system setup is wrong". But that's a waste of time. People continue to associate Python3 and Unicode to annoying bugs, because they don't understand how locales work.
Instead of having to repeat to each user that "hum, maybe your config is wrong", I prefer to support this non convential setup and work as expected ("it just works"). With my latest implementation, setlocale() is only done when LC_CTYPE and LC_NUMERIC are different, which is the corner case which "shouldn't occur in practice".
malemburg commentedon Jan 15, 2018
Sounds like a good compromise :-)
vstinner commentedon Jan 15, 2018
I tested localeconv() with PR 4174 on FreeBSD:
locale.setlocale(locale.LC_ALL, "C")
locale.setlocale(locale.LC_NUMERIC, "ar_SA.UTF-8")
It works as expected, result:
decimal_point: '\u066b'
thousands_sep: '\u066c'
Compare it to Python 3.6 which returns mojibake, it seems like bytes are decoded from Latin1:
decimal_point: '\xd9\xab'
thousands_sep: '\xd9\xac'
Raw byte strings, Python 2.7:
vstinner commentedon Jan 15, 2018
Test on Linux (Fedora 27, glibc 2.26):
It works as expected, result:
decimal_point: '.'
thousands_sep: '\u2009'
Python 3.6 returns mojibake:
decimal_point: '.'
thousands_sep: '\xe2\x80\x89'
Python 2.7 raw strings, thousands_sep = b'\xE2\x80\x89'.
vstinner commentedon Jan 15, 2018
On macOS 10.13.2, I failed to find any non-ASCII decimal_point or thousands_sep in localeconv(). I wrote a script to find all non-ASCII data in all locales:
https://github.com/vstinner/misc/blob/master/python/all_locales.py
vstinner commentedon Jan 15, 2018
New changeset cb064fc by Victor Stinner in branch 'master':
bpo-31900: Fix localeconv() encoding for LC_NUMERIC (bpo-4174)
cb064fc
vstinner commentedon Jan 15, 2018
lc_numeric.py contains a typo, used fixed lc_numeric2.py instead to test my PR 5191 which fixes decimal.Decimal.
vstinner commentedon Jan 15, 2018
New changeset 5f959c4 by Victor Stinner in branch '3.6':
[3.6] bpo-31900: Fix localeconv() encoding for LC_NUMERIC (bpo-4174) (bpo-5192)
5f959c4
andreas-schwab commentedon Jan 28, 2018
That's not true. There is a rich set of *_l functions that take a locale_t object and operate on that locale.
vstinner commentedon Oct 17, 2018
Victor:
Andreas Schwab:
Oh. Do you want to work on a patch to use these functions? If yes, please open a new issue to enhance the code.
vstinner commentedon Nov 28, 2018
See also bpo-28604: localeconv() doesn't support LC_MONETARY encoding different than LC_CTYPE encoding.
vstinner commentedon Nov 28, 2018
The initial bug has been fixed, I close the issue.