Repository navigation
codecs.lookup() ignores non-ASCII characters, whereas encodings.normalize_encoding() copies them #83518
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2020-10-14.15:46:55.467>
created_at = <Date 2020-01-14.21:54:42.124>
labels = ['library', '3.10']
title = 'codecs.lookup() ignores non-ASCII characters, whereas encodings.normalize_encoding() copies them'
updated_at = <Date 2020-10-14.15:55:50.301>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2020-10-14.15:55:50.301>
actor = 'shihai1991'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-14.15:46:55.467>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2020-01-14.21:54:42.124>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 39337
keywords = ['patch']
message_count = 11.0
messages = ['360004', '363645', '364180', '364449', '364452', '364456', '377772', '378281', '378626', '378628', '378630']
nosy_count = 4.0
nosy_names = ['lemburg', 'vstinner', 'serhiy.storchaka', 'shihai1991']
pr_nums = ['18845', '18987', '19069', '22219', '22360', '17997']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue39337'
versions = ['Python 3.10']
Activity
vstinner commentedon Jan 14, 2020
bpo-37751 changed codecs.lookup() in a subtle way: non-ASCII characters are now ignored, whereas they were copied unmodified previously.
I would prefer that codecs.lookup() and encodings.normalize_encoding() behave the same. Either always ignore or always copy.
Moreover, it seems like there is no test on how the encoding names are normalized in codecs.register(). I recall that using codecs.register() in an unit test causes troubles since there is no API to unregister a search function. Maybe we should just add a private function for test in _testcapi.
Serhiy Storchaka wrote an example on my PR:
https://github.com/python/cpython/pull/17997/files
shihai1991 commentedon Mar 8, 2020
How about calling
encodings.normalize_encoding() in codecs.normalizestring()to keep same behavior?(I create PR18845)I can try to add some test cases in next weekend ;)
shihai1991 commentedon Mar 14, 2020
I have try this idea, but it make the testcase of test_io.py failed because some object will call
codecs.Lookup()in__del__().-->extension module will be cleaned before calling__del__().I try to add a
_Py_normalize_unicode_encoding()in unicodeobject.c to support non-ASCII encoding names' normalization(PR18987), but this PR caused many testcases failed.For example:
In master:
python3.9 -c "print('a\xac\u1234\u20ac\u8000\U0010ffff'.encode('iso-8859-15', 'namereplace'))"
result:
b'a\xac\\N{ETHIOPIC SYLLABLE SEE}\xa4\\N{CJK UNIFIED IDEOGRAPH-8000}\\U0010ffff'
after PR18987:
./python -c "print('a\xac\u1234\u20ac\u8000\U0010ffff'.encode('iso-8859-15', 'namereplace'))"
result:
b'a\xac\\N{ETHIOPIC SYLLABLE SEE}\\N{EURO SIGN}\\N{CJK UNIFIED IDEOGRAPH-8000}\\U0010ffff'
shihai1991 commentedon Mar 17, 2020
Oh, there have a problem with this idea:
struct _is is defined in internal/pycore_pystate.h.
_testcapi must test the public Python C API, not CPython internal C API
vstinner commentedon Mar 17, 2020
Use _testinternalcapi in this case.
shihai1991 commentedon Mar 17, 2020
vstinner commentedon Oct 1, 2020
See also bpo-37751 and my PR 17997.
vstinner commentedon Oct 8, 2020
New changeset 3f34237 by Hai Shi in branch 'master':
bpo-39337: Add a test case for normalizing of codec names (GH-19069)
3f34237
vstinner commentedon Oct 14, 2020
New changeset c5b049b by Hai Shi in branch 'master':
bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters (GH-22219)
c5b049b
vstinner commentedon Oct 14, 2020
Thanks Hai Shi for the change and for new codecs and encodings tests.
shihai1991 commentedon Oct 14, 2020
Thanks for everyone's continus review :)