PyUnicode_FromFormatV() doesn't handle non-ascii text correctly

@amauryfa

BPO	9769
Nosy	@amauryfa, @abalkin, @ezio-melotti
Files	pyunicode_fromformat_ascii.patch pyunicode_fromformat_utf8.patch: PyUnicode_FromFormatV patch to use UTF-8

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-10-02.21:12:14.849>
created_at = <Date 2010-09-03.23:52:59.810>
labels = ['interpreter-core', 'type-feature', 'expert-unicode']
title = "PyUnicode_FromFormatV() doesn't handle non-ascii text correctly"
updated_at = <Date 2015-10-02.21:12:14.848>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2015-10-02.21:12:14.848>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2015-10-02.21:12:14.849>
closer = 'vstinner'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2010-09-03.23:52:59.810>
creator = 'vstinner'
dependencies = []
files = ['18800', '27771']
hgrepos = []
issue_num = 9769
keywords = ['patch']
message_count = 15.0
messages = ['115542', '115609', '115820', '115825', '115889', '116045', '116046', '116071', '121561', '121563', '121568', '121582', '121693', '129838', '174080']
nosy_count = 4.0
nosy_names = ['amaury.forgeotdarc', 'belopolsky', 'ezio.melotti', 'reingart']
pr_nums = []
priority = 'low'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue9769'
versions = ['Python 3.5']

I'm trying to document the encoding of all bytes argument of the C API: see bpo-9738. I tried to understand which encoding is used by PyUnicode_FromFormat*() (and PyErr_Format() which calls PyUnicode_FromFormatV()). It looks like ISO-8859-1, see unicodeobject.c near line 1106:

    for (f = format; *f; f++) {
        if (*f == '%') {
            ...
        } else
            *s++ = *f; <~~~~ here
    }

... oh wait, it doesn't work for non-ascii text! Test in gdb:

(gdb) print _PyObject_Dump(PyUnicodeUCS2_FromFormat("iso-8859-1:\xd0\xff"))
object : 'iso-8859-1:\uffd0\uffff'
type : str
refcount: 1
address : 0x83d5d80

b'\xd0\xff' is decoded '\uffd0\xffff' :-( It's a bug.

--

PyUnicode_FromFormatV() should raise an error on non-ascii format character, or decode it correctly as... ISO-8859-1 or something else. It's difficult to support multi byte encodings (like utf-8), ISO-8859-1 is fine. If we choose to raise an error, how can the user format a non-ascii string? Using its_unicode_format.format(...arguments...) or its_unicode_format % arguments? Is it easy to call these methods in C?

2 remarks:

PyUnicode_FromFormat("%s", text) expects a utf-8 buffer.
Very recently (r84472, r84485), some C files of CPython source code were converted to utf-8. And most of the time, the format given to PyUnicode_FromFormat is a string literal.

So it would make sense for PyUnicode_FromFormat to consider the format string as encoded in utf-8. This is worth asking on python-dev though.

PyUnicode_FromFormat("%s", text) expects a utf-8 buffer.

Really? I don't see how "*s++ = *f;" (where s is Py_UNICODE* and f is char*) can decode utf-8. It looks more like ISO-8859-1.

Very recently (r84472, r84485), some C files of CPython source code
were converted to utf-8

Python source code (C and Python) is written in ASCII except maybe some headers or some tests written in Python with #coding:xxx header (or without the header, but in utf-8, for Python3). I don't think that a C file calls PyErr_Format() or PyUnicode_FromFormat(V)() with a non-ascii format string.

> PyUnicode_FromFormat("%s", text) expects a utf-8 buffer.
Really?

The *format* looks more like latin-1, right. But the payload of a "%s" item is decoded as utf-8.

I don't think that a C file calls PyErr_Format() or
PyUnicode_FromFormat(V)() with a non-ascii format string.

At the moment, it's true. My remark is that utf-8 tend to be applied to all kind of files; if someone once decide that non-ascii chars are allowed in (some) string constants, they will be stored in utf-8.

My remark is that utf-8 tend to be applied to all kind of files;
if someone once decide that non-ascii chars are allowed in (some)
string constants, they will be stored in utf-8.

In this case, it will be better to raise an error on non-ascii byte (character) in the format string. It's better to raise an error than to interpret utf-8 as iso-8859-1 (mojibake!). Since nobody noticed this bug (PyFormat_FromString/PyErr_Format expects ISO-8859-1), I suppose that nobody uses non-ASCII format string is always ascii.

Python builtin errors are not localised. If an application uses gettext, I suppose that the error will be raised in the Python code, not in the C API.

Attached patch changes PyFormat_FromStringV (and so PyFormat_FromString and PyErr_Format) to reject non-ascii byte (character) in the format string. I added a test and documented the format string encoding (which is now ASCII). See also bpo-9738 for the documentation about function argument encoding.

@Amaury

@Amaury: Do you agree to reject non-ascii bytes?

TODO: document format encoding in Doc/c-api/*.rst.

Yes, let's be conservative and reject non-ascii bytes in the format string.

Fixed by r84704 in Python 3.2.

I don't understand Victor's argument in msg115889. According to UTF-8 RFC, <http://www.ietf.org/rfc/rfc2279.txt\>:

US-ASCII values do not appear otherwise in a UTF-8 encoded
character stream. This provides compatibility with file systems
or other software (e.g. the printf() function in C libraries) that
parse based on US-ASCII values but are transparent to other
values.

This means that printf-like formatters should not care whether the format string is in UTF-8, Latin1, or any other ASCII-compatible 8-bit encoding. (Passing in multibyte encoding pretending to be bytes would of course lead to havoc, but C type system will protect you from that.)

It is also fairly simple to ssnity-check for UTF-8 if necessary, but in case of PyUnicode_FromFormat, the resulting string will be decoded as UTF-8, so all characters in the format string will be checked anyways.

Am I missing something?

On Friday 19 November 2010 20:42:53 you wrote:

Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment:

I don't understand Victor's argument in msg115889. According to UTF-8 RFC,
<http://www.ietf.org/rfc/rfc2279.txt\>:

US-ASCII values do not appear otherwise in a UTF-8 encoded
character stream. This provides compatibility with file systems
or other software (e.g. the printf() function in C libraries) that
parse based on US-ASCII values but are transparent to other
values.

Most C functions including printf works on multi*byte* strings, not on (wide)
character strings. Whereas PyUnicode_FromFormatV() converts the format string
(bytes) to unicode (characters). If you would like a comparaison in C, it's
like printf()+mbstowcs() in the same function.

This means that printf-like formatters should not care whether the format
string is in UTF-8, Latin1, or any other ASCII-compatible 8-bit encoding.

It's maybe true with bytes input and bytes output (eg. PyString_FromFormatV()
of Python2), but it's no more true with bytes input and str output (eg.
PyUnicode_FromFormatV() of Python3).

It is also fairly simple to ssnity-check for UTF-8 if necessary, but in
case of PyUnicode_FromFormat, the resulting string will be decoded as
UTF-8, so all characters in the format string will be checked anyways.

I choosed to use ASCII instead of UTF-8, because an UTF-8 decoder is long (210
lines) and complex (see PyUnicode_DecodeUTF8Stateful()), whereas ASCII decode
is just: "unicode_char = (Py_UNICODE)byte;" + an if before to check that 0 <=
byte <= 127).

Nobody noticed my change just because the whole Python code base only uses
ASCII argument for the format argument of PyUnicode_FromFormatV().

Victor

On Fri, Nov 19, 2010 at 3:06 PM, STINNER Victor <report@bugs.python.org> wrote:

.. Whereas PyUnicode_FromFormatV() converts the format string
(bytes) to unicode (characters). If you would like a comparaison in C, it's
like printf()+mbstowcs() in the same function.

I see. So it is really the

    else
        *s++ = *f;

that surreptitiously widens the characters.

..

I choosed to use ASCII instead of UTF-8, because an UTF-8 decoder is long (210
lines) and complex (see PyUnicode_DecodeUTF8Stateful()), whereas ASCII decode
is just: "unicode_char = (Py_UNICODE)byte;" + an if before to check that 0 <=
byte <= 127).

I don't think we need 210 lines to replace "*s++ = *f" with proper
UTF-8 logic. Even if we do, the code can be shared with
PyUnicode_DecodeUTF8 and a UTF-8 iterator may be a welcome addition to
Python C API.

On Friday 19 November 2010 21:58:25 you wrote:

> I choosed to use ASCII instead of UTF-8, because an UTF-8 decoder is long
> (210 lines) and complex (see PyUnicode_DecodeUTF8Stateful()), whereas
> ASCII decode is just: "unicode_char = (Py_UNICODE)byte;" + an if before
> to check that 0 <= byte <= 127).

I don't think we need 210 lines to replace "*s++ = *f" with proper
UTF-8 logic. Even if we do, the code can be shared with
PyUnicode_DecodeUTF8 and a UTF-8 iterator may be a welcome addition to
Python C API.

Why should we do that? ASCII format is just fine. Remember that
PyUnicode_FromFormatV() is part of the C API. I don't think that anyone would
use non-ASCII format in C. If someone does that, (s)he should open a new issue
for that :-) But I don't think that we should make the code more complex if
it's just useless.

Victor

On Fri, Nov 19, 2010 at 7:15 PM, STINNER Victor <report@bugs.python.org> wrote:
..

Why should we do that? ASCII format is just fine. Remember that
PyUnicode_FromFormatV() is part of the C API. I don't think that anyone would
use non-ASCII format in C.

Why not. Gettext manual is full of examples with i18nalized format strings.

If someone does that, (s)he should open a new issue
for that :-)

Why new issue? The title of this issue fits perfectly and IMO it is
hard to argue that to "handle non-ascii text correctly" means to raise
an error when non-ascii text is encountered.

I still consider that ASCII format strings should be enough for everyone.

> If someone does that, (s)he should open a new issue for that :-)

Why new issue?

Ok, so I just remove myself from the nosy list.

(moved from issue bpo-16343)

Working in an internationalization proposal <http://python.org.ar/pyar/TracebackInternationalizationProposal\> (issue bpo-16344)
I've stopped at this problem (bpo-9769) where multi byte encodings (like utf-8) is not supported by PyUnicode_FromFormatV()

Beside my proposal, I think utf-8 should be supported for consistency with the other unicode functions, like PyUnicode_FromString() or even unicode_fromformat_arg()

Attached is a patch that:

enhanced the iterator to detect multibyte sequences, with sanity checks about start & continuation bytes
replaced unicode_write_cstr with PyUnicode_DecodeUTF8Stateful
tests

Hope it helps, this is my first patch for cpython and my C skills are a bit rusty, so excuse me if there is any newbie glitch

Navigation Menu

Repository navigation

PyUnicode_FromFormatV() doesn't handle non-ascii text correctly #53978

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

PyUnicode_FromFormatV() doesn't handle non-ascii text correctly #53978

Description

Activity

vstinner commented on Sep 3, 2010

amauryfa commented on Sep 4, 2010

vstinner commented on Sep 7, 2010

amauryfa commented on Sep 7, 2010

vstinner commented on Sep 8, 2010

vstinner commented on Sep 10, 2010

amauryfa commented on Sep 10, 2010

vstinner commented on Sep 11, 2010

abalkin commented on Nov 19, 2010

vstinner commented on Nov 19, 2010

abalkin commented on Nov 19, 2010

vstinner commented on Nov 20, 2010

abalkin commented on Nov 20, 2010

vstinner commented on Mar 2, 2011

reingart commented on Oct 28, 2012

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions