Skip to content

Commit 1cd1fc6

Browse files
gpsheadgpshead
committed
[3.15] gh-129724: gh-156504: Make marshal output independent of reference counts (GH-159192)
The marshal writer decided whether an object can be referenced later (FLAG_REF) from its reference count, so the output for the same value depended on what else in the process held a reference to its parts, and differed between the default and the free-threaded builds' view of the same objects. This is the remaining cause of gh-129724 and the cause of gh-156504. FLAG_REF is now set exactly on the objects which are written more than once in the dump. The writer still makes one pass: it records the position of the type byte of each object it remembers and of each TYPE_REF, and when the dump is complete it sets the flags and renumbers the references. A reference count of one is only used to skip the hashtable. Interned strings are no longer flagged unconditionally. The reader is unchanged and the format version is not bumped: both the old and the new output are loaded by old and new readers. PyMarshal_WriteObjectToFile() now builds the data in memory and writes it with one fwrite(), since the flags are only known at the end. Nothing is written if the object cannot be marshalled. (cherry picked from commit 48f18f0)
1 parent 3ca92d9 commit 1cd1fc6

7 files changed

Lines changed: 789 additions & 108 deletions

File tree

‎Doc/c-api/marshal.rst‎

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -31,29 +31,38 @@ the :py:mod:`Python module documentation <marshal>` for details.
3131
3232
.. note::
3333
34-
In Python 3.15 and earlier, this function does not set the error
35-
indicator if writing to *file* fails.
34+
This function does not set the error indicator if writing to *file*
35+
fails.
3636
The same caveat applies as for :c:func:`PyMarshal_WriteObjectToFile`.
3737
3838
.. c:function:: void PyMarshal_WriteObjectToFile(PyObject *value, FILE *file, int version)
3939
4040
Marshal a Python object, *value*, to *file*.
4141
*version* indicates the file format.
4242
43+
The marshalled representation of *value* is built in memory
44+
and then written to *file*.
45+
Nothing is written if *value* cannot be marshalled.
46+
4347
This function can fail, in which case it sets the error indicator.
4448
Use :c:func:`PyErr_Occurred` to check for that.
4549
4650
.. note::
4751
48-
In Python 3.15 and earlier, this function can write only part of the
49-
data to *file* and return without setting the error indicator, for
50-
example if *value* contains an object which cannot be marshalled or
51-
if writing to *file* fails (see :gh:`155907`).
52-
Code which has to support these versions cannot rely on
53-
:c:func:`PyErr_Occurred` alone to know that all the data was written.
52+
This function can write only part of the data to *file* and return
53+
without setting the error indicator if writing to *file* fails
54+
(see :gh:`155907`).
55+
Code cannot rely on :c:func:`PyErr_Occurred` alone to know that all
56+
the data was written.
5457
It can use :c:func:`PyMarshal_WriteObjectToString` and write the
5558
result itself, or validate what was written.
5659
60+
.. versionchanged:: next
61+
Previously, the data was written to *file* as it was produced,
62+
using a buffer of constant size,
63+
and incomplete data could be written without setting the error
64+
indicator if *value* could not be marshalled.
65+
5766
.. c:function:: PyObject* PyMarshal_WriteObjectToString(PyObject *value, int version)
5867
5968
Return a bytes object containing the marshalled representation of *value*.

0 commit comments

Comments
 (0)