Skip to content

native _json, _pickle, pyexpat, and _elementtree #8741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
youknowone merged 21 commits into from
Sep 19, 2026
Merged

native _json, _pickle, pyexpat, and _elementtree #8741

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
d76be23
_json: add native make_encoder (c_make_encoder)
moreal Sep 7, 2026
4b4bf12
pyexpat: add UseForeignDTD, fix silently swallowed handler exceptions
moreal Sep 8, 2026
58e7dfe
pyexpat: support multi-chunk Parse()/ParseFile() feeds
moreal Sep 8, 2026
4592d32
test: drop stale expectedFailure markers fixed by the pyexpat changes
moreal Sep 8, 2026
a59da40
pyexpat: drive a single background EventReader for true incremental P…
moreal Sep 8, 2026
38795df
pyexpat: add a Sync fallback backend for targets without threads
moreal Sep 8, 2026
5f9c330
_pickle: native Pickler, Unpickler and PickleBuffer
moreal Sep 8, 2026
cfeaf81
_pickle: raise the pickle.find_class audit event
moreal Sep 8, 2026
b5fb603
_pickle: avoid temporary buffers on the save fast paths
moreal Sep 8, 2026
debdcff
pyexpat: make the threaded backend fork-safe
moreal Sep 8, 2026
95fc653
pyexpat: implement name interning via self.intern, matching libexpat
moreal Sep 8, 2026
152dd3f
_elementtree: native Element, SubElement and ParseError
moreal Sep 8, 2026
37e5e0b
_elementtree: native TreeBuilder and XMLParser
moreal Sep 8, 2026
04aee9e
pyexpat: stop rescanning the whole line per position query
moreal Sep 8, 2026
81dc2df
_elementtree: charge Element.__deepcopy__ to the recursion budget
moreal Sep 8, 2026
1e905ad
pyexpat: don't let a stale NeedMore end a Parse() early
moreal Sep 8, 2026
843e1e4
Restore xml.etree test markers instead of skipTest in bodies
youknowone Sep 19, 2026
6ee458f
Clone state before try_to_bool in pickle load_build
youknowone Sep 19, 2026
a440cd4
Drop decorative section-separator comments
youknowone Sep 19, 2026
90d1611
Represent pickle opcodes as an enum
youknowone Sep 19, 2026
40095e5
Use full paths for thread-only types in pyexpat
youknowone Sep 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
_pickle: native Pickler, Unpickler and PickleBuffer 
Adds `crates/stdlib/src/pickle.rs`, a native `_pickle` implementing
Pickler, Unpickler, PickleBuffer, PicklerMemoProxy/UnpicklerMemoProxy,
dump/dumps/load/loads and the PickleError family, so `Lib/pickle.py`
stops falling back to `_Pickler`/`_Unpickler`.

Protocols 0-5 are supported and produce byte-identical output to the
pure-Python pickler, including framing for protocol >= 4, memoization,
`fast` mode, dispatch_table, reducer_override, persistent_id /
persistent_load, find_class with `_compat_pickle` fix_imports, buffer
callbacks and out-of-band protocol 5 buffers. This also turns on
`pickle.PickleBuffer`, which RustPython did not have before.

Speedups over the pure-Python implementation (best of 5, us/op):
pickle 835 -> 21 (39x), pickle_list 525 -> 4.4 (119x),
pickle_dict 4018 -> 32 (125x), unpickle 630 -> 24 (26x),
unpickle_list 317 -> 5.0 (63x). CPython 3.14's C accelerator runs the
same workloads 1.5-2.9x faster than this.

test_pickle now runs 1008 tests (was 470) since the C-implementation
test classes are enabled; test_pickle, test_pickletools,
test_picklebuffer are green and 22 obsolete expectedFailure markers are
removed.

Assisted-by: Claude Code:claude-fable-5-1
  • Loading branch information
@moreal @youknowone
moreal authored and youknowone committed Sep 19, 2026
commit 5f9c3304ac6e0cc520a3a55fc572b56c7c441153
5 changes: 0 additions & 5 deletions Lib/test/pickletester.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2322,7 +2322,6 @@ def test_bad_getattr(self):
for proto in range(2, pickle.HIGHEST_PROTOCOL + 1):
s = self.dumps(x, proto)

@unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'pickle' has no attribute 'PickleBuffer'
def test_picklebuffer_error(self):
# PickleBuffer forbidden with protocol < 5
pb = pickle.PickleBuffer(b"foobar")
Expand Down Expand Up @@ -2898,7 +2897,6 @@ def test_bytearray_memoization(self):
self.assertIsNot(b2a, b2b)
self.assert_is_copy(b2a, b2b)

@unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'pickle' has no attribute 'PickleBuffer'
def test_picklebuffer_memoization(self):
if self.py_version < (3, 8):
self.skipTest('not supported in Python < 3.8')
Expand All @@ -2923,7 +2921,6 @@ def test_picklebuffer_memoization(self):
self.assert_is_copy(b1b, b)
self.assert_is_copy(b2b, b)

@unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'pickle' has no attribute 'PickleBuffer'
def test_empty_picklebuffer_memoization(self):
# gh-148914: Empty writable PickleBuffer memoized an empty bytearray
# with the id of b'' (a singleton in CPython).
Expand Down Expand Up @@ -4306,14 +4303,12 @@ def check_array(arr):
# 2-D, non-contiguous
check_array(arr[::2])

@unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'pickle' has no attribute 'PickleBuffer'
def test_concurrent_mutation_in_buffer_with_bytearray(self):
def factory():
s = b"a" * 16
return bytearray(s), s
self.do_test_concurrent_mutation_in_buffer_callback(factory)

@unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: module 'pickle' has no attribute 'PickleBuffer'
def test_concurrent_mutation_in_buffer_with_memoryview(self):
def factory():
obj = memoryview(b"a" * 32)[10:26]
Expand Down
51 changes: 0 additions & 51 deletions Lib/test/test_pickle.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,13 +51,7 @@ class PyPickleTests(AbstractPickleModuleTests, unittest.TestCase):
Pickler = pickle._Pickler
Unpickler = pickle._Unpickler

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_dump_load_oob_buffers(self):
return super().test_dump_load_oob_buffers()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_dumps_loads_oob_buffers(self):
return super().test_dumps_loads_oob_buffers()


class PyUnpicklerTests(AbstractUnpickleTests, unittest.TestCase):
Expand Down Expand Up @@ -85,17 +79,8 @@ def dumps(self, arg, proto=None, **kwargs):
f.seek(0)
return bytes(f.read())

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_buffer_callback_error(self):
return super().test_buffer_callback_error()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_non_continuous_buffer(self):
return super().test_non_continuous_buffer()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_picklebuffer_error(self):
return super().test_picklebuffer_error()


class PyPicklerTests(AbstractPickleTests, unittest.TestCase):
Expand All @@ -115,33 +100,15 @@ def loads(self, buf, **kwds):
u = self.unpickler(f, **kwds)
return u.load()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_buffers_error(self):
return super().test_buffers_error()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_bytearray_memoization(self):
return super().test_bytearray_memoization()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_bytes_memoization(self):
return super().test_bytes_memoization()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_c_methods(self):
return super().test_c_methods()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_in_band_buffers(self):
return super().test_in_band_buffers()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_oob_buffers(self):
return super().test_oob_buffers()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_oob_buffers_writable_to_readonly(self):
return super().test_oob_buffers_writable_to_readonly()


class InMemoryPickleTests(AbstractPickleTests, AbstractUnpickleTests,
Expand All @@ -162,33 +129,15 @@ def loads(self, buf, **kwds):
test_find_class = None
test_custom_find_class = None

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_buffers_error(self):
return super().test_buffers_error()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_bytearray_memoization(self):
return super().test_bytearray_memoization()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_bytes_memoization(self):
return super().test_bytes_memoization()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_c_methods(self):
return super().test_c_methods()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_in_band_buffers(self):
return super().test_in_band_buffers()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_oob_buffers(self):
return super().test_oob_buffers()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_oob_buffers_writable_to_readonly(self):
return super().test_oob_buffers_writable_to_readonly()


class PersistentPicklerUnpicklerMixin(object):
Expand Down
19 changes: 3 additions & 16 deletions Lib/test/test_picklebuffer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,7 @@
"""

import gc
# TODO: RUSTPYTHON; Implment PickleBuffer
try:
from pickle import PickleBuffer
except ImportError:
PickleBuffer = None
from pickle import PickleBuffer
import weakref
import unittest

Expand All @@ -34,7 +30,6 @@ def check_memoryview(self, pb, equiv):
self.assertEqual(m.format, expected.format)
self.assertEqual(m.tobytes(), expected.tobytes())

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_constructor_failure(self):
with self.assertRaises(TypeError):
PickleBuffer()
Expand All @@ -46,7 +41,6 @@ def test_constructor_failure(self):
with self.assertRaises(ValueError):
PickleBuffer(m)

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_basics(self):
pb = PickleBuffer(b"foo")
self.assertEqual(b"foo", bytes(pb))
Expand All @@ -60,7 +54,6 @@ def test_basics(self):
m[0] = 48
self.assertEqual(b"0oo", bytes(pb))

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_release(self):
pb = PickleBuffer(b"foo")
pb.release()
Expand All @@ -71,7 +64,8 @@ def test_release(self):
# Idempotency
pb.release()

@unittest.expectedFailure # TODO: RUSTPYTHON
# TODO: RUSTPYTHON; PickleBuffer is not traversed by the cycle collector
@unittest.expectedFailure
def test_cycle(self):
b = B(b"foo")
pb = PickleBuffer(b)
Expand All @@ -81,7 +75,6 @@ def test_cycle(self):
gc.collect()
self.assertIsNone(wpb())

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_ndarray_2d(self):
# C-contiguous
ndarray = import_helper.import_module("_testbuffer").ndarray
Expand All @@ -105,20 +98,17 @@ def test_ndarray_2d(self):

# Tests for PickleBuffer.raw()

@unittest.expectedFailure # TODO: RUSTPYTHON
def check_raw(self, obj, equiv):
pb = PickleBuffer(obj)
with pb.raw() as m:
self.assertIsInstance(m, memoryview)
self.check_memoryview(m, equiv)

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_raw(self):
for obj in (b"foo", bytearray(b"foo")):
with self.subTest(obj=obj):
self.check_raw(obj, obj)

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_raw_ndarray(self):
# 1-D, contiguous
ndarray = import_helper.import_module("_testbuffer").ndarray
Expand All @@ -140,13 +130,11 @@ def test_raw_ndarray(self):
equiv = b'\xc8\x01\x00\x00'
self.check_raw(arr, equiv)

@unittest.expectedFailure # TODO: RUSTPYTHON
def check_raw_non_contiguous(self, obj):
pb = PickleBuffer(obj)
with self.assertRaisesRegex(BufferError, "non-contiguous"):
pb.raw()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_raw_non_contiguous(self):
# 1-D
ndarray = import_helper.import_module("_testbuffer").ndarray
Expand All @@ -156,7 +144,6 @@ def test_raw_non_contiguous(self):
arr = ndarray(list(range(12)), shape=(4, 3), format='<i')[::2]
self.check_raw_non_contiguous(arr)

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_raw_released(self):
pb = PickleBuffer(b"foo")
pb.release()
Expand Down
18 changes: 0 additions & 18 deletions Lib/test/test_pickletools.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,33 +62,15 @@ def test_optimize_binput_and_memoize(self):
self.assertIs(unpickled2[1], unpickled2[2])
self.assertNotIn(pickle.BINPUT, pickled2)

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_buffers_error(self):
return super().test_buffers_error()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_bytearray_memoization(self):
return super().test_bytearray_memoization()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_bytes_memoization(self):
return super().test_bytes_memoization()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_c_methods(self):
return super().test_c_methods()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_in_band_buffers(self):
return super().test_in_band_buffers()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_oob_buffers(self):
return super().test_oob_buffers()

@unittest.expectedFailure # TODO: RUSTPYTHON
def test_oob_buffers_writable_to_readonly(self):
return super().test_oob_buffers_writable_to_readonly()


class SimpleReader:
Expand Down
2 changes: 2 additions & 0 deletions crates/stdlib/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ mod mmap;

mod _heapq;
mod _queue;
mod pickle;
mod pyexpat;
mod pystruct;
mod random;
Expand Down Expand Up @@ -248,6 +249,7 @@ pub fn stdlib_module_defs(ctx: &Context) -> Vec<&'static builtins::PyModuleDef>
not(target_os = "android")
))]
posixshmem::module_def(ctx),
pickle::module_def(ctx),
pyexpat::module_def(ctx),
pystruct::module_def(ctx),
_heapq::module_def(ctx),
Expand Down
Loading