Repository navigation
[C API] PEP 782: Add PyBytesWriter API #129813
Description
Activity
- added a commit that references this issue
on Feb 7, 2025 Thank you! This looks great :)
I have a few suggestions for your consideration, purely from the API-surface point of view:
PyAPI_FUNC(void*) PyBytesWriter_WriteBytes( PyBytesWriter *writer, void *buf, const char *bytes, Py_ssize_t size);
Consider
void *bytes(oruint8_t *bytes), per the WIP guidelines.PyAPI_FUNC(void*) PyBytesWriter_Create( PyBytesWriter **writer, Py_ssize_t alloc);
Consider returning the writer, and having
void **bufas an output argument, for consistency with other*_Create/*_Finish/*_Discardfamilies.PyAPI_FUNC(void*) PyBytesWriter_WriteBytes( PyBytesWriter *writer, void *buf, const char *bytes, Py_ssize_t size);
This is meant to be called as
buf = PyBytesWriter_WriteBytes(..., buf, ...), right?
Could it take avoid **bufand update it?
(Same forPyBytesWriter_Format,PyBytesWriter_Extend)PyAPI_FUNC(Py_ssize_t) PyBytesWriter_GetAllocated( PyBytesWriter *writer);
It seems something like
PyBytesWriter_HowMuchMoreCanIWrite(PyBytesWriter *writer, void *buf)would be more useful -- i.e. get the space that's available afterbuf, rather than the total.PyAPI_FUNC(void*) PyBytesWriter_Extend( PyBytesWriter *writer, void *buf, Py_ssize_t extend);
It might be more useful to add
PyBytesWriter_Reserve, with the same signature but ensuring extend bytes after buf are writable.
(If you use thePyBytesWriter_Create&str += PyOS_snprintfpattern you show, you don't really care about any previous overallocation.)For
PyBytesWriter_Extend(), you usually don't want to forget previousPyBytesWriter_Create()andPyBytesWriter_Extend()calls.// allocate 100 bytes buf = PyBytesWriter_Create(&writer, 100); (... write a few bytes ...) // make room to write 5 bytes at the current position // allocate 100+5 bytes (+ overallocation) buf = PyBytesWriter_Extend(writer, buf, 5); (... write more bytes ...)For example,
PyBytes_FromFormatV(format)uses such pattern: it allocatesstrlen(format), and then callPyBytesWriter_Extend()if it's not enough. Copying each format byte is safe since you have at leaststrlen(format)bytes, but some formats need extra space and here comesPyBytesWriter_Extend().Another full example from
_pickle.c:static PyObject * raw_unicode_escape(PyObject *obj) { Py_ssize_t size = PyUnicode_GET_LENGTH(obj); const void *data = PyUnicode_DATA(obj); int kind = PyUnicode_KIND(obj); PyBytesWriter *writer; char *p = PyBytesWriter_Create(&writer, size); if (p == NULL) { return NULL; } for (Py_ssize_t i=0; i < size; i++) { Py_UCS4 ch = PyUnicode_READ(kind, data, i); /* Map 32-bit characters to '\Uxxxxxxxx' */ if (ch >= 0x10000) { /* -1: subtract 1 preallocated byte */ p = PyBytesWriter_Extend(writer, p, 10-1); if (p == NULL) goto error; *p++ = '\\'; *p++ = 'U'; *p++ = Py_hexdigits[(ch >> 28) & 0xf]; *p++ = Py_hexdigits[(ch >> 24) & 0xf]; *p++ = Py_hexdigits[(ch >> 20) & 0xf]; *p++ = Py_hexdigits[(ch >> 16) & 0xf]; *p++ = Py_hexdigits[(ch >> 12) & 0xf]; *p++ = Py_hexdigits[(ch >> 8) & 0xf]; *p++ = Py_hexdigits[(ch >> 4) & 0xf]; *p++ = Py_hexdigits[ch & 15]; } /* Map 16-bit characters, '\\' and '\n' to '\uxxxx' */ else if (ch >= 256 || ch == '\\' || ch == 0 || ch == '\n' || ch == '\r' || ch == 0x1a) { /* -1: subtract 1 preallocated byte */ p = PyBytesWriter_Extend(writer, p, 6-1); if (p == NULL) goto error; *p++ = '\\'; *p++ = 'u'; *p++ = Py_hexdigits[(ch >> 12) & 0xf]; *p++ = Py_hexdigits[(ch >> 8) & 0xf]; *p++ = Py_hexdigits[(ch >> 4) & 0xf]; *p++ = Py_hexdigits[ch & 15]; } /* Copy everything else as-is */ else *p++ = (char) ch; } return PyBytesWriter_Finish(writer, p); error: PyBytesWriter_Discard(writer); return NULL; }
Sometimes, you don't know how many bytes were allocated and you don't care. Just call
PyBytesWriter_Extend(size)to write size bytes. Example:pylong_int_to_decimal_string()inlongobject.c, it gets a writer and just need to extend to write a few bytes.I'm not sure in which case a
PyBytesWriter_Reserve()function would be useful. At least, I don't think that it's currently needed in the Python code base (with my PR). At the same time, I'm not opposed to add it if it's useful :-)Consider returning the writer, and having void **buf as an output argument
I'm afraid of strict aliasing issues.
For example, if Extend() signature is changed to:
PyAPI_FUNC(int) PyBytesWriter_Extend( PyBytesWriter *writer, void **buf, Py_ssize_t extend);
I'm afraid of compiler issues with the cast:
char *buf; (...) if (PyBytesWriter_Extend(writer, &buf, 100) < 0) { /* handle error */ }
char*is cast tovoid*, maybe the compiler will consider that buf cannot be modified byPyBytesWriter_Extend().I had issues similar to that in the past, sadly I forgot details.
For
PyBytesWriter_Extend(), you usually don't want to forget previousPyBytesWriter_Create()andPyBytesWriter_Extend()calls.Ah, I see,
PyBytesWriter_Extendis useful andPyBytesWriter_Reservecan be added later.
As long as the documentation clearly says what the function does, that sounds good.I'm afraid of strict aliasing issues.
I don't see strict aliasing issues here. Were the issues you saw related to casting between
char*andvoid*?PyAPI_FUNC(Py_ssize_t) PyBytesWriter_GetAllocated(
PyBytesWriter *writer);It seems something like PyBytesWriter_HowMuchMoreCanIWrite(PyBytesWriter *writer, void *buf) would be more useful -- i.e. get the space that's available after buf, rather than the total.
I removed
PyBytesWriter_GetAllocated()and I addedPyBytesWriter_GetRemaining(PyBytesWriter *writer, void *buf)instead.I don't see strict aliasing issues here. Were the issues you saw related to casting between char* and void*?
A "recent" example of type punning / strict aliasing issue with Py_CLEAR()/Py_SETREF(): #99701 These macros take a pointer to a pointer to any object (ex:
PyLongObject**) and cast it to a pointer to a pointer to a PyObject (PyObject**).Consider returning the writer, and having void **buf as an output argument, for consistency with other _Create/_Finish/*_Discard families.
It would be inconvenient. If
&bufis passed asvoid**withchar *buf, it fails with a compiler error:error: passing argument 1 of 'PyBytesWriter_Create' from incompatible pointer type [-Wincompatible-pointer-types]You cannot pass
char**asvoid**without an explicit cast tovoid**.Also, as written before, I'm afraid of type punning / strict aliasing issues if
void**is used.Reacted by Petr ViktorinPyAPI_FUNC(void*) PyBytesWriter_WriteBytes(..., const char *bytes, ...)
Consider void *bytes (or uint8_t *bytes), per the WIP guidelines.Ok, I changed bytes parameter type to
void*.I created a discussion: https://discuss.python.org/t/add-pybyteswriter-public-c-api/81182
Reacted by Petr ViktorinI created a discussion: https://discuss.python.org/t/add-pybyteswriter-public-c-api/81182
It seems like most developers are confused by the API which requires to pass writer and buf to most functions. I abandon this API.
105 remaining items
- added a commit that references this issue
on Sep 5, 2026
Feature or enhancement
Proposal:
I propose adding a PyBytesWriter API to create
bytesobjects.PyBytesWriter_Extend()and usage of a "small buffer" of (around) 256 bytesbytesobjects.bytes.API:
Simple example creating the string
b"abc":Example formatting an integer in decimal, the size is not known in advance::
Note: using
PyBytesWriter_Format()would make this code simpler.Example using
PyBytesWriter_Extend(),smilar tobytes.center()with a different API: spaces are number of whitespaces added to the left and to the right:Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
My previous attempt in July/August 2024:
Linked PRs
PyBytesWriterin_json:_match_number_unicode#138957