Repository navigation
Fixed the documentation of the mapping codec APIs #72935
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2017-03-21.08:14:55.008>
created_at = <Date 2016-11-20.10:29:58.188>
labels = ['type-bug', '3.7', 'expert-unicode', 'docs']
title = 'Fixed the documentation of the mapping codec APIs'
updated_at = <Date 2017-03-24.22:09:26.995>
user = 'https://github.com/serhiy-storchaka'bugs.python.org fields:
activity = <Date 2017-03-24.22:09:26.995>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2017-03-21.08:14:55.008>
closer = 'serhiy.storchaka'
components = ['Documentation', 'Unicode']
creation = <Date 2016-11-20.10:29:58.188>
creator = 'serhiy.storchaka'
dependencies = []
files = ['45557', '45623', '45669', '46389']
hgrepos = []
issue_num = 28749
keywords = ['patch']
message_count = 14.0
messages = ['281259', '281629', '281677', '281857', '281864', '281866', '281872', '286027', '286064', '286075', '289484', '290155', '290156', '290166']
nosy_count = 6.0
nosy_names = ['lemburg', 'vstinner', 'ezio.melotti', 'docs@python', 'serhiy.storchaka', 'xiang.zhang']
pr_nums = ['487', '714', '715']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28749'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']
Activity
serhiy-storchaka commentedon Nov 20, 2016
Proposed patch adds the documentation of PyUnicode_Translate() and fixes the documentation of other mapping codec APIs (it is incorrect in Python 3): PyUnicode_DecodeCharmap(), PyUnicode_AsCharmapString(), PyUnicode_EncodeCharmap(), and PyUnicode_TranslateCharmap().
serhiy-storchaka commentedon Nov 24, 2016
Updated patch addresses Xiang's and Victor's comments.
zhangyangyu commentedon Nov 25, 2016
v2 LGTM.
malemburg commentedon Nov 28, 2016
Why are you removing the introductory section on how mappings work ?
Also, this wording needs to be corrected: "bytes (integers in the range from 0 to 255)". Bytes are not integers. I'd suggest to use the more correct wording "bytes strings of length 1".
serhiy-storchaka commentedon Nov 28, 2016
Thanks Xiang. I forgot about comments in headers, the updated path updates them too.
Because it is not correct. I copied it to descriptions of concrete functions with correcting it according to the peculiarity of particular function.
The word "bytes" means here not Python bytes object, but is used in more common meaning: an integer in the range from 0 to 255.
malemburg commentedon Nov 28, 2016
On 28.11.2016 12:24, Serhiy Storchaka wrote:
The only part that is not correct is "single string characters".
This should read "single bytes" or "bytes strings of length 1".
I also don't see where you copied the description. Without some
description of what "mappings" are in the context of the charmap
codec, it's not easy to understand what the purpose of these
APIs is. Please just fix the bytes wording instead of removing the
whole intro.
That's confusing, since we use the term "bytes" as referring
to the bytes object in Python. Please use "integers in the range
0-255".
Aside: The deprecation of PyUnicode_EncodeCharmap() also seems misplaced
in this context, since only the Py_UNICODE version of the API is
deprecated. The functionality still exists and is useful. An API
similar to the _PyUnicode_EncodeCharmap() API should be made publicly
available to accommodate for the deprecation, since the mentioned
PyUnicode_AsCharmapString() and PyUnicode_AsEncodedString()
APIs are not suitable as replacement. PyUnicode_AsCharmapString()
doesn't support error handling (strange, BTW) and
PyUnicode_AsEncodedString() has a completely unrelated meaning (no
idea why it's mentioned here at all).
serhiy-storchaka commentedon Nov 28, 2016
This is not correct. Decoding mappings map not bytes strings, but integers.
And this is not the only incorrect part. Decoding mappings can map to
multicharacter Unicode strings, not to single Unicode characters. Not just
None, but the integer 0xfffe and Unicode string '\ufffe' mean "undefined
mapping".
There are similar incorrectnesses about encoding mappings.
Decoding mappings were desribed in the introduction and in the description of
PyUnicode_DecodeCharmap() (both are outdated and incomplete). I merged and
corrected descriptions and left it only in one place, since
PyUnicode_DecodeCharmap() is the only function that needs this. Same for
encoding mappings. Both decoding and encoding mappings do not have a relation
to PyUnicode_Translate(). The paragraph about a LookupError in the
introduction was totally wrong. I left in the introduction only common part.
Other details are too different in decoding, encoding and translation mappings.
Okay, I'll remove the word "bytes" here. But how would you formulate the
following sentence: "Unmapped bytes (ones which cause a :exc:`LookupError`) as
well as mapped to ``None``, ``0xFFFE`` or ``'\ufffe'`` are treated as "undefined
mapping" and cause an error."?
Only PyUnicode_EncodeCharmap() is deprecated, PyUnicode_AsCharmapString() is
not deprecated. I placed the deprecated function just after its non-deprecated
counerpart following the pattern for other deprecated functions. If you prefer
I'll move both deprecated functions (PyUnicode_EncodeCharmap and
PyUnicode_TranslateCharmap) together at the end of this section.
I don't know why PyUnicode_AsCharmapString() don't support the errors
argument. I added PyUnicode_AsEncodedString() as a replacement (bpo-19569)
because this is the only public non-deprecated way to do a charmap encoding
with errors handling. There is no exact equivalent, but
PyUnicode_AsCharmapString() and PyUnicode_AsEncodedString() cover different
areas of using PyUnicode_EncodeCharmap().
serhiy-storchaka commentedon Jan 22, 2017
How can we move this issue forward? Marc-Andre, have I answered to your objections?
malemburg commentedon Jan 23, 2017
Looking at the implementation, you're right. AFAIR, the first
incarnation of the charmap codec used single chars in Python 2.
I guess the documentation was never updated when the change was
made to use integers instead.
Yes, this was added later on as well. Apparently the docs
were never updated.
Ok, fair enough, let's remove the two paragraphs.
Better:
"""
If *mapping* is *NULL*, Latin-1 decoding will be applied. Else
*mapping* must map bytes ordinals (integers in the range from 0 to 255)
to Unicode strings, integers (which are then interpreted as Unicode
ordinals) or ``None``. Unmapped data bytes - ones which cause a
:exc:`LookupError`, as well as ones which get mapped to ``None``,
``0xFFFE`` or ``'\ufffe'``, are treated as undefined mappings and cause
an error.
"""
No, I'd prefer this deprecation to be undone as long as we
don't have a proper alternative for the API.
Looking at the various deprecations for the Py_UNICODE APIs,
I find that the Unicode API symmetry was severely broken.
In the Python 2 API, we always have an PyUnicode_Encode...() and
corresponding PyUnicode_Decode...() API for every codec.
In Python 3, the encode APIs were apparently all deprecated
due to their use of Py_UNICODE and only the the much less useful
PyUnicode_As...String() APIs were left, which intentionally do not
have an error argument, because they were intended as quick
replacement for PyString_AsString() uses in Python 2.
The only way around this is to have new APIs to reestablish
the previous encoding functionality (only based on
PyObject *unicode instead of Py_UNICODE) at the C API level.
serhiy-storchaka commentedon Jan 23, 2017
"bytes ordinals" is good term. Thank you. Here is an updated patch.
This is different issue.
serhiy-storchaka commentedon Mar 12, 2017
Do you still have objections Marc-Andre?
serhiy-storchaka commentedon Mar 24, 2017
New changeset 88b32eb by Serhiy Storchaka in branch '3.5':
bpo-28749: Fixed the documentation of the mapping codec APIs. (#487) (#715)
88b32eb
serhiy-storchaka commentedon Mar 24, 2017
New changeset 69eab31 by Serhiy Storchaka in branch '3.6':
bpo-28749: Fixed the documentation of the mapping codec APIs. (#487) (#714)
69eab31
serhiy-storchaka commentedon Mar 24, 2017
New changeset c85a266 by Serhiy Storchaka in branch 'master':
bpo-28749: Fixed the documentation of the mapping codec APIs. (#487)
c85a266