Align compiler and VM bytecode contracts with CPython 3.15 by youknowone · Pull Request #9088 · RustPython/RustPython

youknowone · 2026-10-12T11:55:50Z

This PR follows our AI policy

Summary

Align compiler and VM bytecode contracts with official final CPython 3.15.0, pinned to d220409a13b77de7a6c30abe16ea90db01528edc.

This coherent transition couples opcode IDs, caches and stack contracts with iterator/generator creation, common constants, import packing, canonical code storage and marshal field ownership. It also aligns public opcode descriptors and calling behavior, and prevents private constant-specialization IDs from leaking into adaptive exports.

Exactly three compiler-owned Lib files are imported through update_lib, byte-for-byte from final CPython: opcode.py, _opcode_metadata.py, and dis.py. Eight disassembly expectations were migrated from certified output without changing test inputs. No general Lib refresh is included in this compiler-owned diff.

Genuine prerequisite stack

This draft targets the temporary prerequisite branch, not main. The actual merge base is 9e0541bba17e06a70accc21a4f55b794cd1e9d8f, combining:

Pinned main snapshot 9ea138c8891ebf0bdd7865c629d11466d768d2a9. This is the main snapshot used for the local validation below, not the latest main.
Mapping prerequisite source and reviewed hash-error repair at 7d06dd0859685f6f8802a409aaa91e98916663bf. Implement PEP 814 frozendict #8951 was closed without merge; its code remains in this temporary review base solely to preserve the validated compiler context. This draft does not reopen Implement PEP 814 frozendict #8951 or claim that its changes landed.
Serialize bytearray exports with resizing #9087 buffer repair plus the source-less initialization follow-up at 108171daa2657a6c88798a719c76aac68b8d983b. Serialize bytearray exports with resizing #9087 has merged at f163e1371018275e40b4651108818d722b02368c; the additional source-less fix is still unmerged.

Compiler head: a2dacaf4bd7421830318fb413b55b7ab98ba918f, tree 78c286c65cd38565dca93474a34292ee2c66a990.

The triple-dot compiler-only diff is 73 files, 11,688 additions and 1,808 deletions, with only the three Lib files above. Inherited copy.py and prerequisite test changes belong to the actual base. The original compiler checkpoint was followed by the bounded CI corrections described below.

At publication preparation, main is c0694d955cb08c5acee72a3585fd07c5037ea80e; #9085, #9086, #9087 and #9069 have merged. Those later main changes are not claimed as integrated into this validated snapshot.

Please do not merge into this temporary review branch. This draft is for compiler-only review, not a merge-ready claim. Resolve the closed mapping prerequisite’s disposition, reconcile with current main and outstanding follow-ups, then retarget and recheck the resulting merge tree and required hosted checks.

CI corrections and focused validation on the current head

The first hosted run exposed ordinary script validation still assuming LOAD_CONST None, a thread-slot borrow held while exception traceback locals finalized an interpreter, and a public cache-tag contract mismatch. Windows also exposed inconsistent LF/CRLF bytes in the newly added checked-hash cache fixture. The current head fixes those consumers, releases the TLS borrow before exception decrefs, restores the public implementation/version cache tag, and makes the new fixture's source bytes platform-stable. Private 0xDA16 magic and all foreign-format rejection assertions remain unchanged. Existing canonical tests were not changed.

Fresh native build and seven isolated bounded checks passed: the existing 11 returns-only-None subcases; three ordinary traceback-local interpreter cleanup cycles; complete unchanged contextvars, zoneinfo-cache, subinterpreters and sys snippets; and the private cache reader/writer matrix.
The changed private-cache snippet passed through pytest: collection selected one of 510 tests (509 deselected), then exact-node execution collected and passed that one test. No broad snippet execution or ignored tests were run.
Strict root/VM/compiler-core/codegen all-target Clippy passed with warnings denied.
Tested source is the head/tree above; native binary SHA256 is 0c38d8f984194965b3bf549f6f8d213bf48382e071dac921f386528b79fbb3b0. Its VM build-script version string retained 1f1fe04 metadata across the subsequent typed-index compilation correction. The successful build log, final source tree and binary hash are retained separately; an embedded a2dacaf4 version stamp is not claimed.

These are Linux focused results. Windows behavior after the fixture correction and the fresh hosted run remain unverified. The broad workspace/CAPI/doctest/frozen results below belong to the initial head, and are not relabeled as fresh runs of the repair head.

Broad validation of initial head `487ae47f`

The following results were obtained on tree 600b67372dfd18b96f2c48e99e743e09051deb0f before the CI corrections:

Exact CI-feature workspace build and all 23 inventoried lib/bin/integration harnesses: 1,363 passed, zero failures. Separate CAPI: 121 passed, four pre-existing ignored, zero failures.
Doctests: 23 passed, 18 pre-existing ignored; CAPI had none. Existing no_run, compile_fail and ignore annotations were preserved. Actual test-name inventories were checked against the earlier reviewed gate before execution.
Exact workspace MOST_FEATS Clippy and separate CAPI-primary all-target Clippy passed with warnings denied. Selected root/VM/compiler all-target Clippy, normal hooks and all pinned manual opcode hooks also passed.
Fresh default interpreter: 22 of 23 bounded commands passed, including bytearray, bytes, I/O, annotations, dictionary and compiler regressions, 81 copy tests and seven monitoring tests. The full OS snippet stops at its existing chown assertion because this environment returns EINVAL instead of PermissionError; the unchanged official CPython reference fails at the same line and errno. The newly integrated, unchanged mkfifo test definition was run separately and passes. The full OS snippet is not claimed to pass.
Fresh release with frozen stdlib and thin LTO: all 19 retained bounded commands plus five affected checks passed. This includes 81 copy tests and seven monitoring tests. Selected module origins are genuinely frozen, and all 1,746 Python build-input hashes were verified. Compiling all frozen inputs is not execution of every frozen module.
Public metadata: 514 flag rows, 10,794 stack rows and eight disassembly outputs match certified results. All 299 calling outcomes remain unchanged; 22 retain known diagnostic wording differences from CPython.
Unchanged strict 120-case corpus: 117 recursive raw-field matches, zero complete byte matches. Every candidate/reference code, marshal and magic domain and every mismatch path is unchanged from the prior checkpoint. No normalization or reference-bit masking was used.

The current reference interpreter is the official actions/python-versions CPython 3.15.0 build. Its archive/binary identities are recorded; all 120 reference domains were independently verified identical to the earlier source-built final-release oracle. Source-pinned generator checks use the exact final CPython commit above.

Canonical library hashes remain intact. Task-generated private caches were inventoried; remaining files were moved intact outside reachable import paths. Final validation used clean import roots. Source, binaries and complete local evidence were independently reviewed and preserved.

The initial hosted run failed the runtime/fixture cases addressed above and separately encountered three download/setup failures. Hosted checks for the corrective head are pending; no hosted success is claimed.

Remaining compatibility scope

This is a locally validated compiler transition draft, not complete byte-exact CPython compatibility. The private development pyc magic intentionally rejects incompatible intermediate formats; the public cache tag remains implementation/version-derived. Complete marshal parity remains incomplete. Native call ownership, dictionary snapshot ownership, compiler-origin chronology, interning lifetime, frozenset order, explicit lazy-import runtime behavior and the 22 calling diagnostics retain known gaps. PEP 810 runtime support is not implemented. Filename work remains outside this stage. Adaptive export observes individual cells and does not promise a coherent concurrent snapshot.

AI-assisted with OpenAI Codex: implementation, regression tests, review, validation and publication preparation.

Couple opcode/cache/stack metadata with iterator pairs, generator preambles, ordinary import packing, common constants, compiler lowering and VM/JIT consumers. Pin official source/tool hashes and retain strict raw code and marshal comparisons. Use private development magic 0xDA16 and cache tag devabi2 until serialization and Python integration are coherent. Final clean corpus matches recursive code fields in 111/111 requests; full marshal/header exactness remains 0/111. Eight existing disassembly tests remain blocked by unchanged Lib metadata. Monitoring export and frozendict/update_lib integration remain explicit follow-on gates. Validation: compiler-core 23, codegen 807, CAPI 121 (4 existing ignored), focused certified-controller pytest, release/frozen witnesses, workspace and CAPI all-target Clippy. Independent source/artifact review covers manifest ec1013591ae97b4c519f4a2c78a8200664f3479f0c5452991b5fea4d1be669ea. Assisted-by: OpenAI Codex:model identifier unavailable

Capture immutable normalized bytecode before publication and lazily retain one public bytes object. Reconstruct replacement code from canonical instructions. Add cold, warmed, decoded, frozen, monitored and generator-flags coverage. Focused VM check/Clippy, native behavior, certified reference and strict finite corpus validated. Full workspace/CAPI/release follow-on validation remains pending; no serialization parity claim. Assisted-by: OpenAI Codex:model identifier unavailable

Represent runtime constants with one immutable tuple owner and copy-on-write compiler views. Finalize code fields with Context, retain public reader field objects, and preserve canonical normalization without mutating published tuples. Add owner-lifetime, reader-hook and Python identity coverage. Focused checks, Clippy, native tests and finite compiler capture verified. Full integration and marshal byte parity remain pending. Assisted-by: OpenAI Codex:model identifier unavailable

Retain real per-write reference owners and serialize stable code-field objects, including canonical monitored bytecode. Preserve the legacy compiler/frozen path and existing reference selection policy. Focused checks pass; full byte parity, public cache format, native ownership and Lib integration remain open. Assisted-by: OpenAI Codex:model identifier unavailable

Observe physical ownership before common-writer lookup, preserving the exact interned-string exception and all real table and transport owners. Add full versioned stream, container, lifetime and bounded thread-churn regressions. Focused validation improves ordinary exact streams from86 to118 of150; code corpus full parity and native borrowed-call ownership remain unresolved. Assisted-by: OpenAI Codex:model identifier unavailable

Finalize supported constants and five assembly field owners through one private compilation pool. Preserve typed and signed-zero keys, explicit NaN/code/slice exclusions, and frozen/reader/constructor boundaries. Keep real owners and existing marshal policy. Focused checks and bounded behavior pass. Expanded strict corpus retains three pre-existing frozenset-order mismatches; full marshal/public format parity and integration gates remain unresolved. Assisted-by: OpenAI Codex:model identifier unavailable

Combine the independently preserved compiler ABI/storage/writer/cache series with current main, frozendict support, and the bytearray buffer-race repair. Preserve prerequisite Lib imports exactly; no new opcode/dis imports. Combined runtime validation remains pending. Assisted-by: OpenAI Codex:model identifier unavailable

Copied only opcode.py, its _opcode_metadata.py dependency, and dis.py using scripts/update_lib copy-lib from pinned CPython d220409a13b77de7a6c30abe16ea90db01528edc. Local validation stage; Rust prerequisites retain merge-order precedence. Assisted-by: OpenAI Codex:model identifier unavailable

Preserve every test source and assertion. Each updated snapshot body matches both certified final CPython and the integrated candidate with exact final opcode/dis metadata. Retain original snapshots and comparison provenance in the local validation evidence. Assisted-by: OpenAI Codex:model identifier unavailable

…cution Generate final-source public descriptors and stack effects without enabling unsupported execution opcodes. Retain explicit calling-conversion and adaptive-export gaps. Assisted-by: OpenAI Codex:model identifier unavailable

Keep executable and public Rust metadata checks, verify exact Python sources read-only, and cover generator dependencies and outputs in manual hooks. Assisted-by: OpenAI Codex:model identifier unavailable

Preserve binder-first and opcode-first conversion order, accept index operands and explicit None, and validate jump by identity. Keep shared diagnostic wording gaps and platform limits explicit. Assisted-by: OpenAI Codex:model identifier unavailable

Traverse immutable canonical boundaries and preserve arguments and cache cells. Retain public specialization IDs and reject the unsupported legacy iterator encoding. Add ordinary warmed and monitoring regressions. Assisted-by: OpenAI Codex:model identifier unavailable

Merge the exact reviewed PR8951 hash-error repair without changing compiler or canonical library sources. Assisted-by: OpenAI Codex:model identifier unavailable

Merge the genuine prerequisite review base without changing the verified compiler candidate source tree. Historical validation belongs to candidate 2759621, tree 13cd8ae; this commit only records base ancestry. Assisted-by: OpenAI Codex:model identifier unavailable

Preserve the frozen publication checkpoint separately. Integrate main 9ea138c, mapping 7d06dd0, and bytearray 108171d without extending compiler implementation scope. Source-only integration; historical test results are not fresh validation of this tree. Assisted-by: OpenAI Codex:model identifier unavailable

coderabbitai · 2026-10-12T11:55:59Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-12T11:57:17Z

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
[x] lib: cpython/Lib/dis.py
[x] test: cpython/Lib/test/test_dis.py (TODO: 1)

dependencies:

dis

dependent tests: (80 tests)

[ ] lib: cpython/Lib/code.py
[x] test: cpython/Lib/test/test_code_module.py

dependencies:

code (native: builtins, readline, sys)
- argparse (native: sys)
- codeop
- traceback (native: _missing_stdlib_info, _suggestions, collections.abc, importlib.machinery, importlib.resources.readers, importlib.util, itertools, sys, unicodedata)

dependent tests: (2 tests)
- [ ] pdb: test_pdb
- [ ] sqlite3.main: test_sqlite3

[x] test: cpython/Lib/test/test_compile.py
[x] test: cpython/Lib/test/test_compiler_assemble.py
[x] test: cpython/Lib/test/test_compiler_codegen.py
[x] test: cpython/Lib/test/test_peepholer.py

dependencies:

dependent tests: (no tests depend on compile)

[x] lib: cpython/Lib/opcode.py
[x] lib: cpython/Lib/_opcode_metadata.py
[x] test: cpython/Lib/test/test__opcode.py
[x] test: cpython/Lib/test/test_opcodes.py

dependencies:

opcode

dependent tests: (50 tests)

opcode: test__opcode test_code test_compile test_dis test_peepholer
- dis: test_ast test_compiler_assemble test_dtrace test_fstring test_inspect test_monitoring test_opcache test_patma test_positional_only_arg test_type_cache
  - bdb: test_bdb test_pdb
  - inspect: test_abc test_argparse test_asyncgen test_buffer test_builtin test_clinic test_collections test_coroutines test_decimal test_enum test_functools test_generators test_grammar test_ntpath test_operator test_posixpath test_pydoc test_signal test_sqlite3 test_traceback test_turtle test_type_annotations test_type_params test_types test_typing test_unittest test_yield_from test_zipimport test_zipimport_support test_zoneinfo
  - modulefinder: test_importlib test_modulefinder
  - trace: test_trace

[x] lib: cpython/Lib/warnings.py
[x] lib: cpython/Lib/_py_warnings.py
[ ] test: cpython/Lib/test/test_warnings (TODO: 1)

dependencies:

warnings

dependent tests: (55 tests)

warnings: test_argparse test_array test_asyncio test_buffer test_builtin test_codecs test_codeop test_compile test_coroutines test_ctypes test_decimal test_descr test_fnmatch test_fstring test_funcattrs test_genericpath test_glob test_global test_grammar test_gzip test_hashlib test_hmac test_importlib test_inspect test_io test_logging test_ntpath test_os test_pickle test_posix test_pty test_pyclbr test_pyrepl test_random test_re test_runpy test_set test_socket test_sqlite3 test_str test_string_literals test_support test_sys test_sys_settrace test_tarfile test_tempfile test_thread test_threading test_type_cache test_typing test_unicode_file_functions test_unittest test_unparse test_xml_etree test_zipimport

[ ] test: cpython/Lib/test/test_format.py

dependencies:

dependent tests: (no tests depend on format)

[ ] test: cpython/Lib/test/test_import (TODO: 3)

dependencies:

dependent tests: (no tests depend on import)

[ ] test: cpython/Lib/test/test_monitoring.py (TODO: 3)

dependencies:

dependent tests: (no tests depend on monitoring)

[x] test: cpython/Lib/test/test_generators.py
[ ] test: cpython/Lib/test/test_genexps.py (TODO: 2)
[x] test: cpython/Lib/test/test_generator_stop.py
[x] test: cpython/Lib/test/test_yield_from.py

dependencies:

dependent tests: (no tests depend on generator)

Legend:

[+] path exists in CPython
[x] up-to-date, [ ] outdated

Recognize common None loads when checking script returns. Keep the public cache tag version-derived while retaining the private DA16 format boundary. Release the thread-slot borrow before exception decrefs can finalize an interpreter. Assisted-by: OpenAI Codex:model identifier unavailable

Write explicit LF for filesystem source so checked-hash ZIP inputs match the same fixture bytes on every platform. Preserve all cache validity and foreign-format rejection assertions. Assisted-by: OpenAI Codex:model identifier unavailable

Assisted-by: OpenAI Codex:model identifier unavailable

…l 3.15 Migrate only CI-red canonical test modules and their required syntax-warning fixture through update_lib from CPython d220409a. Preserve generated migration marker output. Clear created generator frames using actual owner state, publish closed state before warning callbacks, and expose the final read-only gi_state contract. Assisted-by: OpenAI Codex:model identifier unavailable

Retain public co_name subclass owners for comparison and hashing. Match omitted warning modules against filenames through the exact final CPython helper imported with update_lib. Port final constant sequence folding order and align the string-format diagnostics exposed by canonical CI. Assisted-by: OpenAI Codex:model identifier unavailable

Check the existing optional _testcapi import before allocating monitoring tool IDs, so setUp SkipTest leaves no state for subsequent tests. Assisted-by: OpenAI Codex:model identifier unavailable

Preserve the 31-string fixture and require one ordered tuple constant load, one GET_ITER, and no streaming list builders, matching final CPython optimizer output. Assisted-by: OpenAI Codex:model identifier unavailable

Allow final exception-event masks locally and remove only the local event registration on DISABLE. Match final bytes unsupported-letter diagnostics. Copy the three affected canonical test definitions through update_lib from pinned final CPython. Assisted-by: OpenAI Codex:model identifier unavailable

codspeed · 2026-10-13T01:52:08Z

Merging this PR will not alter performance

✅ 62 untouched benchmarks
⏩ 4 skipped benchmarks¹

_{Comparing youknowdot:compiler-315-current-integration (246c309) with main (85d5956)²}

4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
No successful run was found on compiler-315-current-prerequisites (9e0541b) during the generation of this report, so main (85d5956) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

Clear the existing thread-local suppression flag when a handler receives an exception or an exception escapes a frame, independently of optional monitoring event masks. Assisted-by: OpenAI Codex:model identifier unavailable

Preserve canonical CI results while collecting contained list/dict manager diagnostics and logs for the remaining Windows failure. Remove this temporary instrumentation once localized. Assisted-by: OpenAI Codex: model identifier unavailable

Keep the two ordinary proxy methods and existing Job Object containment; include source-extracted canonical lifecycle stages to localize the remaining worker cleanup failure. Finalizer tracebacks remain diagnostic failures even when the canonical helper suppresses exceptions. Assisted-by: OpenAI Codex: model identifier unavailable

youknowdot added 16 commits October 12, 2026 00:20

Pin opcode checks to final Python 3.15 without rewriting canonical Lib

6e2a962

Keep executable and public Rust metadata checks, verify exact Python sources read-only, and cover generator dependencies and outputs in manual hooks. Assisted-by: OpenAI Codex:model identifier unavailable

Reconcile reviewed dictionary error handling with compiler candidate

2759621

Merge the exact reviewed PR8951 hash-error repair without changing compiler or canonical library sources. Assisted-by: OpenAI Codex:model identifier unavailable

youknowdot added 8 commits October 12, 2026 21:13

Use typed constant index conversion in script validation

a2dacaf

Assisted-by: OpenAI Codex:model identifier unavailable

Skip optional monitoring C API tests before tool registration

246c309

Check the existing optional _testcapi import before allocating monitoring tool IDs, so setUp SkipTest leaves no state for subsequent tests. Assisted-by: OpenAI Codex:model identifier unavailable

Assert final constant-list iterable folding in compiler regression

7ecedcf

Preserve the 31-string fixture and require one ordered tuple constant load, one GET_ITER, and no streaming list builders, matching final CPython optimizer output. Assisted-by: OpenAI Codex:model identifier unavailable

youknowdot added 3 commits October 13, 2026 11:06

Navigation Menu

Repository navigation

Align compiler and VM bytecode contracts with CPython 3.15 - #9088

Align compiler and VM bytecode contracts with CPython 3.15#9088
youknowone wants to merge 27 commits into
RustPython:compiler-315-current-prerequisitesfrom
youknowdot:compiler-315-current-integration

youknowone commented Oct 12, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 12, 2026 •

edited

Loading

Draft PR not reviewed

Uh oh!

github-actions Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Oct 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

youknowone commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Genuine prerequisite stack

CI corrections and focused validation on the current head

Broad validation of initial head 487ae47f

Remaining compatibility scope

Uh oh!

coderabbitai Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Draft PR not reviewed

Uh oh!

github-actions Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Library Dependencies

Uh oh!

codspeed Bot commented Oct 13, 2026

Merging this PR will not alter performance

Footnotes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

youknowone commented Oct 12, 2026 •

edited

Loading

Broad validation of initial head `487ae47f`

coderabbitai Bot commented Oct 12, 2026 •

edited

Loading

github-actions Bot commented Oct 12, 2026 •

edited

Loading