Skip to content

gh-148706: Pure-Python pickle._Pickler can be significantly optimized #148707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mjbommar wants to merge 9 commits into from
Closed

gh-148706: Pure-Python pickle._Pickler can be significantly optimized #148707

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
94b53eb
pickle: add exact-container fast paths to pure-Python _batch_appends …
mjbommar Apr 17, 2026
bb9d721
pickle: inline frame-check and short-circuit common atomic types in s…
mjbommar Apr 17, 2026
69eed04
pickle: add Misc/pickle-perf-diary.md + raw bench data
mjbommar Apr 17, 2026
285fcae
pickle: fix large-dict mutation semantics; reorder save() fast paths …
mjbommar Apr 17, 2026
7c6af84
pickle: cache BININT1 opcode bytes for n in 0..255 (F4)
mjbommar Apr 17, 2026
2f1d38b
pickle: inline the MEMOIZE proto-4+ path in memoize() (F2)
mjbommar Apr 17, 2026
e917108
pickle: extend save() fast path to bytes (F6)
mjbommar Apr 17, 2026
024a57c
pickle: update perf diary for round 2 (F1-F6) + raw data
mjbommar Apr 17, 2026
4f0b43c
gh-148706: add NEWS entry for pickle._Pickler optimization
mjbommar Apr 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
pickle: extend save() fast path to bytes (F6) 
Add `bytes` to the save() dispatch fast path, placed AFTER the four
non-memoized atomics (int / None / bool / float) so int-heavy workloads
don't pay an extra type-identity branch before hitting their direct
dispatch. bytes is memoized, so memo.get is inlined alongside the
save_bytes call the same way str is.

Dump deltas on a bytes-heavy workload:

    list_of_short_bytes_5k    dump -10.7%
    list_of_medium_bytes_1k   dump -11.7%
    dict_bytes_to_int_2k      dump  -9.1%
    list_of_bytearrays_1k     dump  +0.4%   (bytearray isn't caught;
                                             hits dispatch table)

Main-bench impact (F2 -> F6v2), within ±0.7% across all benches.
bytearray specialisation is a separate question; rare in real payloads.

test_pickle 1060/1060.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
  • Loading branch information
@mjbommar @claude
mjbommar and claude committed Apr 17, 2026
commit e917108ace45f92c3ce80d0ad4907bbd22f56fe7
10 changes: 10 additions & 0 deletions Lib/pickle.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -610,6 +610,8 @@ def save(self, obj, save_persistent_id=True):
self.save_str(obj)
return
# int / None / bool / float: not memoized; skip memo.get entirely.
# Placed before bytes so int-heavy workloads don't pay an extra
# branch miss before hitting their fast path.
if t is int:
self.save_long(obj)
return
Expand All @@ -622,6 +624,14 @@ def save(self, obj, save_persistent_id=True):
if t is float:
self.save_float(obj)
return
# bytes: memoized; same inline memo pattern as str.
if t is bytes:
x = self.memo.get(id(obj))
if x is not None:
self.write(self.get(x[0]))
return
self.save_bytes(obj)
return

# Check the memo (non-atomic, non-str types)
x = self.memo.get(id(obj))
Expand Down