Repository navigation
mbcs encoding ignores errors #39624
Description
| BPO | 850997 |
|---|---|
| Nosy | @loewis, @mhammond, @vstinner, @devdanzin, @tjguk, @ezio-melotti |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2010-06-18.23:24:39.064>
created_at = <Date 2003-11-29.01:24:21.000>
labels = ['type-feature', 'expert-unicode', 'OS-windows']
title = 'mbcs encoding ignores errors'
updated_at = <Date 2010-06-18.23:24:39.057>
user = 'https://github.com/mhammond'bugs.python.org fields:
activity = <Date 2010-06-18.23:24:39.057>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2010-06-18.23:24:39.064>
closer = 'vstinner'
components = ['Unicode', 'Windows']
creation = <Date 2003-11-29.01:24:21.000>
creator = 'mhammond'
dependencies = []
files = ['1122', '1123', '17618', '17635', '17690']
hgrepos = []
issue_num = 850997
keywords = ['patch']
message_count = 17.0
messages = ['19177', '19178', '19179', '19180', '82015', '82133', '106277', '106278', '106407', '107513', '107517', '107611', '107612', '107957', '107965', '108018', '108149']
nosy_count = 6.0
nosy_names = ['loewis', 'mhammond', 'vstinner', 'ajaksu2', 'tim.golden', 'ezio.melotti']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue850997'
versions = ['Python 3.2']
Activity
mhammond commentedon Nov 29, 2003
The following snippet:
Should raise a UnicodeError. The errors param is
completely ignored, and the function always works as
though errors='replace'.
Attaching a test case, and the start of a patch. The
patch has a number of issues:
I have handled 'strict', 'ignore' and 'replace' -
however, 'ignore' and 'replace' currently are exactly
the same (ie, replace)
character failed in the conversion. Thus, the
exception I raise implies the first character is the
one that failed. For the same reason, I have made no
attempt to support error callbacks.
Comments/guidance appreciated.
mhammond commentedon Nov 29, 2003
Logged In: YES
user_id=14198
Attaching a patch. This patch also attempts to handle
Encode, but I haven't worked out how to exercise this
code-path - ie, what mbcs encoded string can I pass that can
not be converted to unicode?
As I mentioned, patch has a few issues
theller commentedon Nov 29, 2003
Logged In: YES
user_id=11105
No idea why this was assigned to me - unicode is certainly
not one of my strengths.
loewis commentedon Dec 1, 2003
Logged In: YES
user_id=21627
The conventional semantics of "ignore" would be "remove the
failing characters from the output". This would be difficult
to implement if the Microsoft API provides no detailed error
indication.
You could try to get more detailed error indication by
re-encoding the resulting string with a NULL buffer,
counting the number of characters that have successfully
been encoded, atleast in the .decode case.
In the .encode case, you could try using \0 as the default
char. To my knowledge, no ACP ever uses \0 in a multi-byte
string.
What is the meaning of the WC_DEFAULTCHAR flag, in
WideCharToMultiByte, and why are you not using it?
I'm somewhat concerned with backwards compatibility, since
the mbcs codec has never returned errors. So this should be
applied to 2.4 only, and listed in whatsnew.tex.
devdanzin commentedon Feb 14, 2009
Is this behavior still present? If so, is it still interesting to change it?
mhammond commentedon Feb 14, 2009
It is still present, but I'm not sure what problems can be seen due to
this so can't comment on its desirability. It would also introduce a
backwards compatability concern but I've not enough experience to know
how much of a problem that would be in practice either.
vstinner commentedon May 22, 2010
I patched py3k with mbcs_errors.patch (only encode_mbcs, not the decoder function) and most test pass: I opened bpo-8784 for test_tarfile failure.
I don't think that it's a problem that mbcs only supports few error handlers, eg. 'strict', 'replace' and 'errors' (but not 'ignore' nor 'surrogateescape'). mbcs should be avoided anyway :-) It is kept for backward compatibility (with Python2). Python3 tries to avoid it by using the Unicode functions of Windows API.
I don't know exactly where mbcs is still used in Python3. If mbcs becomes more strict and raise new errors, I would like to say that the problem comes from the program, not in the encodig, and the program should be fixed (especilly if the "program" is the Python standard library).
About the backward compatibility with Python < 3.2: I don't know exactly if this change would be a problem or not. I bet that few people use (directly or indirectly) mbcs with Python 3.1 (on Windows), and few peple (or nobody) would notice this change. And as I wrote, if someone notices a problem: the problem should be fixed in the function using mbcs, not in the codec.
vstinner commentedon May 22, 2010
Since this change breaks backward compatibility, it's a very bad idea to change mbcs codec in Python 2.7: remove this version from this issue.
vstinner commentedon May 24, 2010
Updated version of the patch for py3k:
The whole test suite pass with these patch.
vstinner commentedon Jun 11, 2010
I worked again on the patch. I opened new issues to prepare the new mbcs codec:
bpo-8967 can be used to get the translated message of a mbcs encode error. PyErr_GetWindowsMessage() returns a PyUnicodeObject, whereas make_translate_exception() and PyUnicodeTranslateError_SetReason() expect a "char*". Another patch is requied: translate_reason_unicode.patch (attached to this issue, not tested). But I don't think that the message is very important for now :-)
bpo-8784 (tarfile/Windows: Don't use mbcs as the default encoding) is still open.
vstinner commentedon Jun 11, 2010
New version of the patch:
The patch requires bpo-8969 patch (use mbcs in strict mode to encode/decode filenames).
vstinner commentedon Jun 12, 2010
Tim: are you interested in testing this patch?
vstinner commentedon Jun 12, 2010
Update the patch (I commited the patch on tarfile module): version 3.
vstinner commentedon Jun 16, 2010
Patch version 4:
vstinner commentedon Jun 16, 2010
I commited the last patch to py3k: r82037. Let see how the buildbots react :-)
tjguk commentedon Jun 17, 2010
I'm unlikely to get to it soon. If there's no urgency I can
look at it later. FWIW, it's not something I'm especially
familiar with.
On 12/06/2010 01:02, STINNER Victor wrote:
vstinner commentedon Jun 18, 2010
Close this issue: nothing special on the buildbots.