Repository navigation
pickle protocol 5: writable in-band PickleBuffer with empty tobytes() can memoize b'' via BYTEARRAY8, so later bytes references unpickle as bytearray #148914
Description
Bug report
Bug description:
The problem
- for an in-band, writable
PickleBuffer, the pickler doesbuf = m.tobytes(). That value is always bytes - if the buffer is empty, buf is always the
b''singleton, soid(buf)is the same for every such call - the pickler then does
in_memo = id(buf) in self.memo - so once
b''has been memoized anywhere earlier in the same dump, every later empty in-bandtobytes()in that dump sees in_memo is true - so any later part of the same pickle that is supposed to store
b''as bytes may instead reuse that memo. The unpickler then returns the bytearray, not bytes.
Code that requires bytes (strict isinstance etc.) can fail during unpickling when it receives a bytearray.
Excerpt from pickler.py, but the same applies to the C implementation:
if in_memo:
pb_branch = "_save_bytearray_no_memo"
self._save_bytearray_no_memo(buf)
else:
pb_branch = "save_bytearray"
self.save_bytearray(buf)One minimal repro example is:
import dill
from pickle import PickleBuffer
# a bit artificial example to trigger wrong flow
def repro_minimal() -> None:
pb = PickleBuffer(memoryview(bytearray()))
def f():
pass
blob = dill.dumps((pb, f), protocol=5)
# it fails when trying to pass a bytearray instead of bytes to dill's _create_code
dill.loads(blob)
print("dill.loads ok")Less synthetic if you have Pandas and PyArrow
import pandas as pd
def repro_arrow_empty_dataframe() -> None:
col = "EMPTY_STRING_COLUMN"
df = pd.DataFrame({col: pd.Series([""], dtype="string[pyarrow]")})
def g():
pass
blob = dill.dumps((df, g), protocol=5)
dill.loads(blob)
print("dill.loads ok")Both result in TypeError: code() argument 16 must be bytes, not bytearray
However, that code() case is only one example; the underlying issue is bytes vs bytearray memo reuse, and similar failures can appear anywhere bytes are required.
CPython versions tested on:
3.13
Operating systems tested on:
macOS
Linked PRs
- gh-148914: Fix memoization of in-band PickleBuffer in the Python implementation #149052
- [3.14] gh-148914: Fix memoization of in-band PickleBuffer in the Python implementation (GH-149052) #149274
- [3.13] gh-148914: Fix memoization of in-band PickleBuffer in the Python implementation (GH-149052) #149275
Metadata
Metadata
Assignees
Projects
- Status
Activity
serhiy-storchaka commentedon Apr 26, 2026
Sorry, I don't understand what the problem is. If
m.readonlyis false andin_memois true,_save_bytearray_no_memo(buf)is called, which writes the BYTEARRAY8 opcode without writing MEMOIZE and memoizingbuf. What is wrong with this?Do you have an example that does not use third-party libraries?
madkrouger commentedon Apr 27, 2026
Hey,
I think the bug is that bytes
b''can unpickle as a bytearray if the memo forid(b'')was first created as a bytearray:I believe treating
b''as bytes would be safer and more compatible with callers that expect empty bytesserhiy-storchaka commentedon Apr 27, 2026
Thank you for your example @madkrouger. I see that the problem actually is opposite to your initial description. If
in_memois false, the code for empty bytearray is written, and it is memoized asb''. Now the following references tob''(without PickleBuffer) will be unpickled as the same empty bytearray.There was also other bug here, for all PickleBuffers. PickleBuffers were incorrectly memoized, so identical PickleBuffers did not preserve identity after unpickling. This was only in the Python implementation.
#149052 fixes this, now the Python implementation produces the same data as the C implementation.
pythongh-148914: Fix memoization of in-band PickleBuffer in the Pytho…
madkrouger commentedon Apr 28, 2026
Great, thanks for taking care of this!
gh-148914: Fix memoization of in-band PickleBuffer in the Python impl…
[3.14] gh-148914: Fix memoization of in-band PickleBuffer in the Pyth…
[3.13] gh-148914: Fix memoization of in-band PickleBuffer in the Pyth…