Optimize _PyUnicodeWriter implementation

To prepare gh-119182 implementation, I propose to optimize first the _PyUnicodeWriter implementation. For example, optimize the UTF-8 decoder in PyUnicode_FromFormat() by avoiding the creation of a temporary buffer: write directly characters in the writer.

Linked PRs

pythongh-119396: Optimize %S format of PyUnicode_FromFormat()

I do not know how much this is needed, but the code looks correct.

Could you please collect some data about types used with %S and %R? And how often %S and %R are used with and without the width and the precision? Just add writing the type name in the special log file and run the test suite, then count most common names.

Could you please collect some data about types used with %S and %R?

Sure, here you have (I didn't check width or precision):

Details

%R, str: 113038 (80%)
%S, str: 24825 (18%)
%R, bool: 770 (1%)
%R, tuple: 696 (0%)
%R, int: 507 (0%)
%R, list: 393 (0%)
%S, int: 136 (0%)
%S, _GenericAlias: 120 (0%)
%R, bytes: 116 (0%)
%R, dict: 113 (0%)
%R, NoneType: 85 (0%)
%R, frame: 62 (0%)
%R, type: 52 (0%)
%R, types: 43 (0%)
%R, function: 42 (0%)
%R, _io: 33 (0%)
%S, types: 30 (0%)
%R, datetime: 24 (0%)
%R, object: 19 (0%)
%R, float: 18 (0%)
%R, _StoreAction: 16 (0%)
%R, Element: 15 (0%)
%R, _CallableGenericAlias: 12 (0%)
%R, typing: 12 (0%)
%S, NoneType: 11 (0%)
%R, _asyncio: 11 (0%)
%R, QueryTestCase: 10 (0%)
%S, type: 10 (0%)
%A, NoneType: 7 (0%)
%R, FixedOffset: 7 (0%)
%R, EnumType: 6 (0%)
%A, str: 6 (0%)
%S, _AnnotatedAlias: 6 (0%)
%R, NewType: 6 (0%)
%R, CustomPyPicklerClass: 6 (0%)
%R, ArrayBinASCIITest: 5 (0%)
%R, Mock: 5 (0%)
%R, _contextvars: 5 (0%)
%S, object: 4 (0%)
%S, Exception: 4 (0%)
%R, Foo: 4 (0%)
%S, _ProtocolMeta: 4 (0%)
%R, _thread: 4 (0%)
%R, complex: 4 (0%)
%R, Task: 4 (0%)
%R, _ALWAYS_EQ: 3 (0%)
%R, collections: 3 (0%)
%R, builtin_function_or_method: 3 (0%)
%R, Test: 3 (0%)
%S, _CallableGenericAlias: 3 (0%)
%R, functools: 3 (0%)
%R, CPartialSubclass: 3 (0%)
%R, Future: 3 (0%)
%R, Derived: 3 (0%)
%R, StopIteration: 2 (0%)
%S, BadStr: 2 (0%)
%R, code: 2 (0%)
%S, dict: 2 (0%)
%S, NotImplementedType: 2 (0%)
%S, ellipsis: 2 (0%)
%S, Cheese: 2 (0%)
%S, SubBytes: 2 (0%)
%S, list: 2 (0%)
%S, _TypedDictMeta: 2 (0%)
%S, MutatesYourDict: 2 (0%)
%R, Meta: 2 (0%)
%R, SSLSocket: 2 (0%)
%R, CustomStr: 2 (0%)
%S, ValueError: 2 (0%)
%R, C: 2 (0%)
%R, socket: 2 (0%)
%R, TarInfo: 2 (0%)
%R, sub: 1 (0%)
%R, MyFileIO: 1 (0%)
%R, Completer: 1 (0%)
%R, aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa: 1 (0%)
%R, StackObject: 1 (0%)
%R, bytearray: 1 (0%)
%R, CustomBytes: 1 (0%)
%R, CustomByteArray: 1 (0%)
%R, memoryview: 1 (0%)
%R, array: 1 (0%)
%R, posix: 1 (0%)
%R, xml: 1 (0%)
%R, sqlite: 1 (0%)
%R, Base: 1 (0%)
%R, expr: 1 (0%)

For str, int and float:

%R, str: 113038 (80%)
%S, str: 24825 (18%)
%R, int: 507 (0%)
%S, int: 136 (0%)
%R, float: 18 (0%)
%A, str: 6 (0%)

Patch:

diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 480b671390..d2df7bf62d 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -2750,6 +2750,7 @@ unicode_fromformat_arg(_PyUnicodeWriter *writer,
         PyObject *obj = va_arg(*vargs, PyObject *);
         PyObject *str;
         assert(obj);
+fprintf(stderr, "@@@ PyUnicode_FromFormat(%%S): %s\n", Py_TYPE(obj)->tp_name);
         str = PyObject_Str(obj);
         if (!str)
             return NULL;
@@ -2766,6 +2767,7 @@ unicode_fromformat_arg(_PyUnicodeWriter *writer,
         PyObject *obj = va_arg(*vargs, PyObject *);
         PyObject *repr;
         assert(obj);
+fprintf(stderr, "@@@ PyUnicode_FromFormat(%%R): %s\n", Py_TYPE(obj)->tp_name);
         repr = PyObject_Repr(obj);
         if (!repr)
             return NULL;
@@ -2782,6 +2784,7 @@ unicode_fromformat_arg(_PyUnicodeWriter *writer,
         PyObject *obj = va_arg(*vargs, PyObject *);
         PyObject *ascii;
         assert(obj);
+fprintf(stderr, "@@@ PyUnicode_FromFormat(%%A): %s\n", Py_TYPE(obj)->tp_name);
         ascii = PyObject_ASCII(obj);
         if (!ascii)
             return NULL;

Thank you. According to these results we can ignore int and float and only optimize str. Of course, the data obtained in such way is very rough estimation of real world usage, but the difference is too large.

Unfortunately, %R for str is not optimized in this PR, and the usage of str with %S is not overwhelming, so this optimization can have very minor effect. I am not sure now that the gain is worth the cost.

I also write PR gh-119398 for this issue: "Optimize PyUnicode_FromFormat() UTF-8 decoder".

gh-119396: Optimize PyUnicode_FromFormat() UTF-8 decoder (#119398)

pythongh-119396: Optimize unicode_repr()

gh-119396: Optimize unicode_repr() (#119617)

gh-119396: Optimize unicode_decode_utf8_writer() (#119957)

pythongh-119396: Optimize unicode_decode_utf8_writer() (python#119957)

pythongh-119396: Optimize PyUnicode_FromFormat() UTF-8 decoder (pytho…

pythongh-119396: Optimize unicode_repr() (python#119617)

pythongh-119396: Optimize unicode_decode_utf8_writer() (python#119957)

Navigation Menu

Repository navigation

Optimize _PyUnicodeWriter implementation #119396

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Optimize _PyUnicodeWriter implementation #119396

Description

Linked PRs

Activity

serhiy-storchaka commented on May 22, 2024

vstinner commented on May 22, 2024

serhiy-storchaka commented on May 22, 2024

vstinner commented on May 22, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions