[C API] PEP 782: Add PyBytesWriter API

Feature or enhancement

Proposal:

I propose adding a PyBytesWriter API to create bytes objects.

Efficient API thanks to overallocation in PyBytesWriter_Extend() and usage of a "small buffer" of (around) 256 bytes
Avoid creating incomplete/inconsistent bytes objects.
Avoid mutating immutable bytes.

API:

typedef struct PyBytesWriter PyBytesWriter;

PyAPI_FUNC(void*) PyBytesWriter_Create(
    PyBytesWriter **writer,
    Py_ssize_t alloc);
PyAPI_FUNC(void) PyBytesWriter_Discard(
    PyBytesWriter *writer);
PyAPI_FUNC(PyObject*) PyBytesWriter_Finish(
    PyBytesWriter *writer,
    void *buf);

PyAPI_FUNC(Py_ssize_t) PyBytesWriter_GetRemaining(
    PyBytesWriter *writer,
    void *buf);
PyAPI_FUNC(void*) PyBytesWriter_Extend(
    PyBytesWriter *writer,
    void *buf,
    Py_ssize_t extend);
PyAPI_FUNC(void*) PyBytesWriter_WriteBytes(
    PyBytesWriter *writer,
    void *buf,
    const void *bytes,
    Py_ssize_t size);
PyAPI_FUNC(void*) PyBytesWriter_Format(
    PyBytesWriter *writer,
    void *buf,
    const char *format,
    ...);

Simple example creating the string b"abc":

PyObject* create_abc(void)
{
    PyBytesWriter *writer;
    char *str = PyBytesWriter_Create(&writer, 3);
    if (writer == NULL) return NULL;

    memcpy(str, "abc", 3);
    str += 3;

    return PyBytesWriter_Finish(writer, str);
}

Example formatting an integer in decimal, the size is not known in advance::

PyObject* format_int(int value)
{
    PyBytesWriter *writer;
    char *str = PyBytesWriter_Create(&writer, 20);
    if (writer == NULL) return NULL;

    str += PyOS_snprintf(str, 20, "%i", value);

    return PyBytesWriter_Finish(writer, str);
}

Note: using PyBytesWriter_Format() would make this code simpler.

Example using PyBytesWriter_Extend(),smilar to bytes.center() with a different API: spaces are number of whitespaces added to the left and to the right:

static PyObject *
byteswriter_center_example(Py_ssize_t spaces, char *str, Py_ssize_t str_size)
{
    PyBytesWriter *writer;
    char *buf = PyBytesWriter_Create(&writer, spaces * 2);
    if (buf == NULL) {
        goto error;
    }
    assert(PyBytesWriter_GetRemaining(writer, buf) == spaces * 2);

    // Add left spaces
    memset(buf, ' ', spaces);
    buf += spaces;
    assert(PyBytesWriter_GetRemaining(writer, buf) == spaces);

    // Copy string
    buf = PyBytesWriter_Extend(writer, buf, str_size);
    if (buf == NULL) {
        goto error;
    }
    assert(PyBytesWriter_GetRemaining(writer, buf) == spaces + str_size);

    memcpy(buf, str, str_size);
    buf += str_size;
    assert(PyBytesWriter_GetRemaining(writer, buf) == spaces);

    // Add right spaces
    memset(buf, ' ', spaces);
    buf += spaces;
    assert(PyBytesWriter_GetRemaining(writer, buf) == 0);

    return PyBytesWriter_Finish(writer, buf);

error:
    PyBytesWriter_Discard(writer);
    return NULL;
}

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

My previous attempt in July/August 2024:

Linked PRs

pythongh-129813: Add PyBytesWriter C API (version 2)

pythongh-129813: Add PyBytesWriter C API (with size flavor)

pythongh-129813: Add PyBytesWriter C API (flavor with size)

pythongh-129813, PEP 782: Add PyBytesWriter C API

gh-129813, PEP 782: Add PyBytesWriter C API (#138822)

pythongh-129813, PEP 782: Add PyBytesWriter_Format()

pythongh-129813, PEP 782: Use PyBytesWriter in binascii

pythongh-129813, PEP 782: Use PyBytesWriter in posix extension

pythongh-129813, PEP 782: Use Py_GetConstant(Py_CONSTANT_EMPTY_BYTES)

pythongh-129813, PEP 782: Use PyBytesWriter in socket and mmap

gh-129813, PEP 782: Add PyBytesWriter_Format() (#138824)

pythongh-129813, PEP 782: Use PyBytesWriter in lzma and zlib

pythongh-129813, PEP 782: Use PyBytesWriter in pickle and struct

pythongh-129813, PEP 782: Use PyBytesWriter in _hashopenssl

pythongh-129813, PEP 782: Use PyBytesWriter in memoryview

pythongh-129813, PEP 782: Use PyBytesWriter in _PyBytes_FromList()

pythongh-129813, PEP 782: Use PyBytesWriter in _PyBytes_DecodeEscape2()

gh-129813, PEP 782: Use PyBytesWriter in _hashopenssl (#138835)

pythongh-129813, PEP 782: Use PyBytesWriter in _PyBytes_FormatEx()

pythongh-129813, PEP 782: Use PyBytesWriter in bytes_concat()

gh-129813, PEP 782: Use PyBytesWriter in binascii (#138825)

gh-129813, PEP 782: Use PyBytesWriter in pickle and struct (#138833)

gh-129813, PEP 782: Use PyBytesWriter in _PyBytes_DecodeEscape2() (#1…

gh-129813, PEP 782: Use Py_GetConstant(Py_CONSTANT_EMPTY_BYTES) (#138830

gh-129813, PEP 782: Use PyBytesWriter in socket and mmap (#138831)

gh-129813, PEP 782: Use PyBytesWriter in memoryview (#138836)

gh-129813, PEP 782: Use PyBytesWriter in posix extension (#138829)

gh-129813, PEP 782: Use PyBytesWriter in _PyBytes_FromList() (#138837)

gh-129813, PEP 782: Use PyBytesWriter in lzma and zlib (#138832)

pythongh-129813, PEP 782: Use PyBytesWriter in utf8_encoder()

pythongh-129813, PEP 782: Use PyBytesWriter in _io.FileIO.readall

gh-129813, PEP 782: Use PyBytesWriter in _PyBytes_FormatEx() (#138839)

pythongh-129813, PEP 782: Use PyBytesWriter in _codecs.escape_decode()

pythongh-129813, PEP 782: Use PyBytesWriter in _curses

pythongh-129813, PEP 782: Use PyBytesWriter in fcntl

pythongh-129813, PEP 782: Use PyBytesWriter in _hashopenssl

pythongh-129813, PEP 782: Use PyBytesWriter in _sha3

pythongh-129813, PEP 782: Init small_buffer in PyBytesWriter_Create()

gh-129813, PEP 782: Use PyBytesWriter in fcntl (#138921)

gh-129813, PEP 782: Init small_buffer in PyBytesWriter_Create() (#138924

gh-129813, PEP 782: Use PyBytesWriter in _curses (#138920)

gh-129813, PEP 782: Use PyBytesWriter in _codecs.escape_decode() (#13…

pythongh-129813, PEP 782: Use PyBytesWriter in _ssl

pythongh-129813, PEP 782: Use PyBytesWriter in _winapi.PeekNamedPipe()

pythongh-129813, PEP 782: Add PyBytesWriter.overallocate

gh-129813, PEP 782: Use PyBytesWriter in _hashopenssl (#138922)

gh-129813, PEP 782: Use PyBytesWriter in _winapi.PeekNamedPipe() (#13…

gh-129813, PEP 782: Use PyBytesWriter in _io.FileIO.readall (#138901)

gh-129813, PEP 782: Use PyBytesWriter in _sha3 (#138923)

pythongh-129813, PEP 782: Use PyBytesWriter in bufferedio.c

pythongh-129813, PEP 782: Use PyBytesWriter in FileIO.read()

gh-129813, PEP 782: Use PyBytesWriter in FileIO.read() (#138955)

gh-129813, PEP 782: Use PyBytesWriter in bufferedio.c (#138954)

pythongh-129813, PEP 782: Use PyBytesWriter in _sqlite

gh-129813, PEP 782: Add PyBytesWriter.overallocate (#138941)

pythongh-129813, PEP 782: Soft deprecate _PyBytes_Resize()

gh-129813, PEP 782: Soft deprecate _PyBytes_Resize() (#138964)

pythongh-129813, PEP 782: Add doc reference link

gh-129813, PEP 782: Add doc reference link (#138986)

pythongh-129813, PEP 782: Use PyBytesWriter in _Py_bytes_maketrans()

pythongh-129813, PEP 782: Use PyBytesWriter in _multiprocessing

pythongh-129813, PEP 782: Use PyBytesWriter in _testclinic

pythongh-129813, PEP 782: Set invalid bytes in PyBytesWriter

gh-129813, PEP 782: Use PyBytesWriter in _Py_bytes_maketrans() (#139044)

gh-129813, PEP 782: Use PyBytesWriter in _multiprocessing (#139047)

gh-129813, PEP 782: Use PyBytesWriter in _testclinic (#139048)

gh-129813, PEP 782: Set invalid bytes in PyBytesWriter (#139054)

pythongh-129813, PEP 782: Use PyBytesWriter in _socket

pythongh-129813, PEP 782: Optimize byteswriter_resize()

gh-129813, PEP 782: Use PyBytesWriter in _socket (#139097)

gh-129813, PEP 782: Optimize byteswriter_resize() (#139101)

pythongh-129813, PEP 782: Use PyBytesWriter in ssl.MemoryBIO

gh-129813, PEP 782: Use PyBytesWriter in ssl.MemoryBIO (#139113)

gh-129813, PEP 782: Use PyBytesWriter in _sqlite (#138956)

pythongh-129813, PEP 782: Use PyBytesWriter in bufferedio.c

gh-129813, PEP 782: Use PyBytesWriter in _ssl (#138929)

gh-129813, PEP 782: Use PyBytesWriter in bufferedio.c (#139121)

pythongh-129813, PEP 782: Use PyBytesWriter in socket recvmsg()

gh-129813, PEP 782: Use PyBytesWriter in socket recvmsg() (#139131)

pythongh-129813, PEP 782: Fix refleak in sock_recvfrom()

gh-129813, PEP 782: Fix refleak in sock_recvfrom() (GH-139151)

gh-129813, PEP 782: Use PyBytesWriter in utf8_encoder() (#138874)

pythongh-129813, PEP 782: Remove private _PyBytesWriter API

pythongh-129813, PEP 782: Remove the private _PyBytesWriter API

gh-129813, PEP 782: Remove the private _PyBytesWriter API (#139264)

I converted most functions using PyBytes_FromStringAndSize(NULL, size) and _PyBytes_Resize() to the new public PyBytesWriter API. I also converted all functions using the old private _PyBytesWriter API and I removed this private API.

pythongh-129813: Fix PyBytesWriter tests

gh-129813: Fix PyBytesWriter tests (#139892)

pythongh-129813: Enhance PyBytesWriter documentation

gh-129813: Enhance PyBytesWriter documentation (#140152)

pythongh-129813: Enhance PyBytesWriter documentation (python#140152)

pythongh-129813: Document that PyBytesWriter_GetData() cannot fail

gh-129813: Document that PyBytesWriter_GetData() cannot fail (#145900)

pythongh-129813: Document that PyBytesWriter_GetData() cannot fail (p…

gh-129813: Fix typo in byteswriter_resize() (#156938)

[3.15] gh-129813: Fix typo in byteswriter_resize() (GH-156938) (#156979)

pythongh-129813: Check size in PyBytesWriter_FinishWithSize()

gh-129813: Check size in PyBytesWriter_FinishWithSize() (#157226)

pythongh-129813: Fix typo in byteswriter_resize() (python#156938)

pythongh-129813: Check size in PyBytesWriter_FinishWithSize() (python…

[3.15] gh-129813: Check size in PyBytesWriter_FinishWithSize() (#157226…

Navigation Menu

Repository navigation

[C API] PEP 782: Add PyBytesWriter API #129813

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[C API] PEP 782: Add PyBytesWriter API #129813

Description

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

Activity

vstinner commented on Sep 24, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions