Repository navigation
disallow the "surrogatepass" handler for non utf-* encodings #58124
Description
| BPO | 13916 |
|---|---|
| Nosy | @loewis, @vstinner, @ezio-melotti, @serhiy-storchaka |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/vstinner'
closed_at = None
created_at = <Date 2012-01-31.23:51:10.468>
labels = ['type-feature', 'expert-unicode']
title = 'disallow the "surrogatepass" handler for non utf-* encodings'
updated_at = <Date 2014-08-17.15:19:53.801>
user = 'https://bugs.python.org/kennyluck'bugs.python.org fields:
activity = <Date 2014-08-17.15:19:53.801>
actor = 'serhiy.storchaka'
assignee = 'vstinner'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2012-01-31.23:51:10.468>
creator = 'kennyluck'
dependencies = []
files = ['35257', '35259', '35262', '35264']
hgrepos = []
issue_num = 13916
keywords = ['patch']
message_count = 20.0
messages = ['152416', '159520', '159525', '159528', '218597', '218600', '218601', '218602', '218603', '218605', '218611', '218612', '218613', '218615', '218617', '218655', '218658', '218659', '218740', '225448']
nosy_count = 6.0
nosy_names = ['loewis', 'vstinner', 'ezio.melotti', 'python-dev', 'kennyluck', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13916'
versions = ['Python 3.5']
Activity
kennyluck commentedon Jan 31, 2012
Currently the "surrogatepass" handler always encodes the surrogates in UTF-8 and hence the behavior for, say, "\udc80".encode("latin-1", "surrogatepass").decode("latin-1") might be unexpected and I don't even know what would, say, "\udc80\udc80".encode("big5", "surrogatepass").decode("big5"), return. Regardless of the fact that the documentation says "surrogatepass" is specific to utf-8", the currently behavior is arguably not too harmful thanks to PyBytesObject's '\0' ending (so that ((p[0] & 0xf0) == 0xe0 || (p[1] & 0xc0) == 0x80 || (p[2] & 0xc0) == 0x80) in PyCodec_SurrogatePassErrors would not crash).
However, I suggest we have the system either 1) raise early LookupError 2) raise the original Unicode(Decode|Encoding)Exception as soon as PyCodec_SurrogatePassErrors is called. I prefer the former.
Having this could shorten PyCodec_SurrogatePassErrors significantly in the patch I will shortly submit for issue bpo-12892 as all the error conditions for utf-8, utf-16 and utf-32 are predicable* and almost all the conditionals could be removed. (The * statement is arguable if someone initializes interp->codec_search_path before _PyCodecRegistry_Init and the utf-16/32 encoders are overwritten. I don't think we need to worry about this too much though. Or am I wrong here?)
loewis commentedon Apr 28, 2012
I fail to see the problem. If the error handler does not produce meaningful results in some context, then just don't use it.
The whole point of error handlers is that they handle errors; using them shouldn't ever cause errors/exceptions.
serhiy-storchaka commentedon Apr 28, 2012
The problem is that "surrogatepass" specific to utf-8 and there is no standard way to decode alone surrogates in utf-16.
With utf-32 this "works" only thanks to the bug -- utf-32 allows alone surrogates (issue bpo-12892).
If the "surrogatepass" worked with utf-16 and utf-32, it would be natural to throw ValueError for other encodings.
loewis commentedon Apr 28, 2012
I see. The proper reaction for a codec that can't handle a certain error then is to raise the original exception. I'm -1 on raising LookupError when trying to find the error handler - this would suggest that the error handler does not exist, which is not true.
As for simplifying the implementation: it might be reasonable to special-case surrogatepass inside the individual codecs, rather than looking up the error handler. Then the error handler could just be identical to "strict", except that UTF-8, UTF-16, and UTF-32 individually special-case this error handler in their encoders and decoders.
serhiy-storchaka commentedon May 15, 2014
This issue was mainly resolved in bpo-12892. The surrogatepass error handler now works with UTF-16* and UTF-32* encodings. But for other encodings it behaves as for UTF-8 (preserve old behavior). Should we change the behavior for non-UTF encodings end raise an exception (as for "strict")?
serhiy-storchaka commentedon May 15, 2014
Here is a patch which disallows the surrogatepass handler for non-utf encodings. Please test it on Windows.
vstinner commentedon May 15, 2014
Serhiy Storchaka wrote:
I don't see your patch.
serhiy-storchaka commentedon May 15, 2014
Oh, sorry.
loewis commentedon May 15, 2014
LGTM
python-dev commentedon May 15, 2014
New changeset 5e98a50e0f55 by Serhiy Storchaka in branch 'default':
Issue bpo-13916: Disallowed the surrogatepass error handler for non UTF-*
http://hg.python.org/cpython/rev/5e98a50e0f55
16 remaining items
StanFromIreland commentedon Jul 12, 2025
Is this still a valid issue? I do not see any Windows buildbot failing this test?
pythongh-58124: Avoid CP_UTF8 in UnicodeDecodeError
vstinner commentedon Aug 5, 2025
I wrote #137415 to change that. It's based on my old cp_encoding_name.patch attached to this issue.
gh-58124: Avoid CP_UTF8 in UnicodeDecodeError (#137415)
pythongh-58124: Avoid CP_UTF8 in UnicodeDecodeError (pythonGH-137415)
pythongh-58124: Avoid CP_UTF8 in UnicodeDecodeError (pythonGH-137415)
[3.13] gh-58124: Avoid CP_UTF8 in UnicodeDecodeError (GH-137415) (#13…
vstinner commentedon Aug 7, 2025
I close the issue. The last point (CP_UTF8) has been addressed.
pythongh-58124: Avoid CP_UTF8 in UnicodeDecodeError (python#137415)
[3.14] gh-58124: Avoid CP_UTF8 in UnicodeDecodeError (GH-137415) (#13…