Repository navigation
punycode codec ignores the error handler argument #56472
Description
| BPO | 12263 |
|---|---|
| Nosy | @loewis, @vstinner, @bitdancer |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2013-12-17.01:52:17.251>
created_at = <Date 2011-06-04.23:16:17.897>
labels = []
title = 'punycode codec ignores the error handler argument'
updated_at = <Date 2013-12-17.01:52:17.245>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2013-12-17.01:52:17.245>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2013-12-17.01:52:17.251>
closer = 'vstinner'
components = []
creation = <Date 2011-06-04.23:16:17.897>
creator = 'vstinner'
dependencies = []
files = ['22373']
hgrepos = []
issue_num = 12263
keywords = ['patch']
message_count = 8.0
messages = ['137666', '137667', '138401', '138498', '138771', '138773', '138801', '206392']
nosy_count = 3.0
nosy_names = ['loewis', 'vstinner', 'r.david.murray']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue12263'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']
Activity
vstinner commentedon Jun 4, 2011
b'abc\xff'.decode('punycode', 'ignore') raises the same error than b'abc\xff'.decode('punycode', 'replace') or b'abc\xff'.decode('punycode'): it uses the strict error handler, and simply ignores the error handler.
punycodec, as idna, should raise an exception if the error handler is different than strict.
vstinner commentedon Jun 4, 2011
base64_codec.py uses an assertion to check the error handler: it should use an if+raise (assertions are ignored in optimized mode, python -O).
vstinner commentedon Jun 15, 2011
punycode_errors.patch:
I don't think that the change should be documented, because punycode has no section in Doc/library/codecs.rst, and I hope that nobody uses something different than strict :-)
vstinner commentedon Jun 17, 2011
@martin: Can you review my patch?
loewis commentedon Jun 21, 2011
What's the point of disallowing the replace error handler? That's a slightly incompatible change, isn't it?
vstinner commentedon Jun 21, 2011
Oh, I forgot to give a little bit more details.
b'abc\xff-'.decode('punycode', 'ignore') and b'abc\xff-'.decode('punycode', 'replace') raise a UnicodeDecodeError: the error handler is just useless (ignored) here.
With my patch, b'abc\xff-'.decode('punycode', 'ignore') gives 'abc'. (If I change the code to accept replace) b'abc\xff-'.decode('punycode', 'replace') gives also 'abc', but 'replace' doesn't work correctly in the part after "-" contain illegal byte sequences.
For example, b'a\xff-\xffb\xffga\xff'.decode("punycode", "replace") gives 'a�', whereas I would expect 'a�é' or 'aé�'. b'a-bga\xff'.decode("punycode", "replace") gives 'aé' as b'a-bga'.decode("punycode", "replace"), whereas I would expect 'aé�' or something like that.
It's just that I'm unable to patch punycode decoder to support the replace handler. Do you want to "implement" it?
I don't think so because I consider that the punycode decoder never supported error handlers (other than strict) in Python 3.
What do you think?
loewis commentedon Jun 21, 2011
That's not my point:
b"foo".decode("punycode","replace")
currently succeeds, but raises an UnicodeError under the patch.
vstinner commentedon Dec 17, 2013
I prefer to not touch punycode right now, it works, there is no need to modify it.