Skip to content

Begin the Python 3.15 target and essential-library migration - #8950

Draft
youknowdot wants to merge 8 commits into
RustPython:mainfrom
youknowdot:cpython-315-essential-bootstrap
Draft

Begin the Python 3.15 target and essential-library migration#8950
youknowdot wants to merge 8 commits into
RustPython:mainfrom
youknowdot:cpython-315-essential-bootstrap

Conversation

@youknowdot

@youknowdot youknowdot commented Oct 2, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

Begin the Python 3.15 migration by changing the interpreter target to 3.15.0alpha and bringing up the essential libraries against CPython v3.15.0rc2 (435c9e5a798c99653e3ab64ce29baed0e4f3dfee). The interpreter boots with normal site initialization and with -S, using the new frozen importlib and the matching source library tree.

This draft includes the pending native prerequisites from #8945 (sentinel), #8946 (eager _imp hook), #8947 (compile(module=...)) and #8948 (atomic module-cache reordering). The target/library commits follow those prerequisites; no broader standard-library or complete compiler port is included.

  • Update target/version metadata and the CPython comparison pin
  • Migrate site, _sitebuiltins, pkgutil, importlib (including metadata/resources), test.support, test_importlib, test_import and their required fixtures from the pinned source
  • Preserve existing RustPython test markers using scripts/update_lib; remove obsolete upstream test locations and retain support helpers still used by unmigrated 3.14 tests
  • Preserve native atomic cache movement at both surviving 3.15 importlib reorder sites and use the canonical builtin import callback
  • Match the new _imp.create_builtin name validation and missing-builtin error behavior, with native/CPython differential regression coverage
  • State explicitly that Python 3.15 compatibility is in progress

Remaining compatibility work

  • PEP 810 lazy imports still execute eagerly. Site uses explicitly marked function-local imports to retain its startup import boundary
  • PEP 814 is being implemented separately. Two private readonly test-support tables currently use an explicitly marked MappingProxyType adaptation
  • Bytecode magic stays at the actually implemented format (3627); the metadata change does not claim full 3.15 bytecode or C-API compatibility
  • The new ._pth no-user-site case retains one precise expectedFailure for existing unsupported ._pth configuration. No blanket new skip or assertion changes were added
  • Broader 3.15 language/runtime work, including the independently tracked PEP 798 gap, remains outside this initial bootstrap

The generated upgrade-pylib automation source and lock are unchanged. Their pre-existing pin/compiler drift needs a separate generator update. Only the standalone documentation-update workflow's comparison default changes here.

Validation

Actual release startup, target/cache tag, frozen atomic/hierarchical importlib, all native prerequisite APIs and source-Lib paths were verified. The six native regression snippets pass, including the new builtin-loader validation; their CPython 3.15 comparisons pass as well.

  • test_site test_support test_importlib test_pkgutil test_import test_compile test_ast test_warnings test_dict test_ordered_dict: 2,545 tests run, 195 skipped, all ten suites successful
  • Threaded import-entrypoint matrix: all 90 scenarios pass across import statements, builtin __import__ and import_module
  • Repeated cross-package/hierarchical import stress: 1,000 iterations each, no failures
  • Full snippets with CPython 3.15 comparator: 469 passed / 23 failed. Thirteen failures are known Unix-socket/chown sandbox limits or a bounded CPython stress timeout; nine are unchanged 3.14 oracle expectations under CPython 3.15. The native missing-builtin change also fails that old expectation. The existing missing-builtin assertion is intentionally retained unchanged until Python 3.15 is released, as requested by the maintainer. No skip or expected-failure marker is added for it; both its CPython and native failures remain visible. This is not a green full-suite claim.
  • Integrated Rust workspace tests: 1,348 passed, 18 ignored. Separate C-API tests: 115 passed, 4 ignored. Both Clippy stages passed without warnings. Source hashes matched before and after; the executed VM test binary has the expected final commit marker and new native features.
  • Configured commit hooks and changed-workflow zizmor scan: passed (offline scan, five existing suppressions)

The earlier unmigrated test_import run failed five times because upstream 3.15 no longer sets module __cached__. Migrating the matching tests/fixture resolved those errors. Its new warning-filtering test passes; the new builtin-loader validation requirement is implemented directly rather than marked as an expected failure.

At publication, GitHub CI and performance workflows require maintainer approval. The existing dependency job succeeds but reads the base branch's 3.14.7 pin after checking out only PR Lib files; its green status does not validate the new 3.15 dependency graph. The local essential suites above use the pinned 3.15.0rc2 libraries.

AI assistance

Implemented, reviewed and tested with OpenAI Codex under the requesting RustPython maintainer's direction. No human code review or maintainer privilege is implied for this account. The runtime does not expose an exact model version; commits use Assisted-by: Codex:model-version-unavailable rather than inventing one.

@youknowdot
Implement the sentinel builtin from PEP 661
b994412 
Add unique sentinel values with copy/pickle identity, caller module capture, union support, and cycle collection. Compare constructor behavior and six upstream tests with CPython v3.15.0rc2; retain the existing Python target and bundled libraries.

Assisted-by: Codex:model-version-unavailable
@youknowdot
Add the eager-runtime importlib lazy-attribute hook
2847724 
Expose _imp._set_lazy_attributes(modobj, name, /), used by CPython 3.15 importlib after eager loading. Match its empty pending-lazy-module registry behavior, including name type and string-subclass hashing checks. This does not implement lazy imports or change the target version or bundled libraries.

Assisted-by: Codex:model-version-unavailable
@youknowdot
Support compile module context across AST warning paths
733591c 
Add Python 3.15 compile(module=...) warning filtering. Move format-spec diagnostics to a fallible, literal-aware AST boundary; preserve nested literal warnings while excluding raw text and comments, map function-type fragments to source locations, and suppress duplicate type-comment emissions with per-call tracking.

Keep unrelated legacy warning-detection and exact diagnostic metadata gaps documented by differential probes. Add focused flag, source-location, exception, reentrancy, filtering and literal-boundary regression coverage.

Assisted-by: Codex:model-version-unavailable
@youknowdot
Reorder the module cache atomically during imports
1c5f81c 
Add a private exact-dict move primitive and importlib dispatcher so shutdown-order updates never temporarily remove a module. Preserve current values and custom-mapping fallback behavior; validate reentrant equality probe witnesses, invalidate layout caches, and compact moved entries.

Cover all four existing bootstrap reorder sites, including legacy-loader cleanup. Add deterministic trace-window, loader replacement/removal, reentrancy, cache, iterator and compaction regressions. The private native primitive uses true-move semantics; it does not promise arbitrary pop/set callback equivalence.

Assisted-by: Codex:model-version-unavailable
@youknowdot
Begin the Python 3.15 essential-library bootstrap
ee04a69 
Target Python 3.15.0alpha and import site, pkgutil, importlib and supporting test libraries from CPython v3.15.0rc2 (435c9e5a798c99653e3ab64ce29baed0e4f3dfee). Keep native atomic module-cache movement at the surviving importlib reorder sites and use the canonical builtin import callback. Preserve explicit temporary eager-import and readonly-table adaptations while their full PEP implementations remain separate. This starts the migration and does not claim complete 3.15 bytecode or runtime compatibility.

Assisted-by: Codex:model-version-unavailable
@youknowdot
Update import tests and builtin loader validation for Python 3.15
5fc8341 
Import pinned rc2 test_import and its fixtures, preserving RustPython markers. The upstream tests remove obsolete module.__cached__ expectations. Match create_builtin validation for missing, non-string, empty, non-ASCII and unknown module names; unknown names must not succeed just because an unrelated module is cached. Add a CPython differential regression.

Assisted-by: Codex:model-version-unavailable
@youknowdot
Preserve the validated string key when creating builtins
6b0a494 
Use the PyStr reference directly after validating the ASCII name, fixing the compile error from the previous PyUtf8Str-only accessor.

Assisted-by: Codex:model-version-unavailable
@youknowdot
Describe Python 3.15 as the migration target
57eb80b 
Make the remaining compatibility work explicit in the project overview rather than implying that the initial target change completes support.

Assisted-by: Codex:model-version-unavailable
@coderabbitai

coderabbitai Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
[ ] lib: cpython/Lib/site.py
[ ] lib: cpython/Lib/_sitebuiltins.py
[ ] test: cpython/Lib/test/test_site.py (TODO: 5)

dependencies:

  • site (native: _io, _pyrepl.main, _pyrepl.pager, _pyrepl.readline, _pyrepl.unix_console, _pyrepl.windows_console, atexit, builtins, errno, readline, sitecustomize, sys, usercustomize)
    • _sitebuiltins
    • _sitebuiltins (native: _pyrepl.pager, sys)
    • warnings (native: _contextvars, _thread, _warnings, builtins, sys)
    • locale, os, pydoc, rlcompleter, stat, textwrap, traceback

dependent tests: (1 tests)

  • site: test_site

[ ] lib: cpython/Lib/pkgutil.py
[ ] test: cpython/Lib/test/test_pkgutil.py

dependencies:

  • pkgutil (native: importlib.machinery, importlib.util, marshal, os.path, sys)
    • importlib (native: _abc, _adapters, _bootstrap, _bootstrap_external, _collections, _common, _frozen_importlib, _frozen_importlib_external, _functional, _functools, _imp, _io, _itertools, _meta, _text, _warnings, collections.abc, email.message, importlib.abc, importlib.metadata, importlib.readers, itertools, marshal, nt, posix, resources.readers, resources.simple, sys, winreg)
    • collections, functools, inspect, os, re, zipimport

dependent tests: (11 tests)

  • pkgutil: test_pkgutil test_pydoc test_pyrepl test_runpy
    • pydoc: test_enum
      • pdb: test_pdb
      • xmlrpc.server: test_docxmlrpc test_xmlrpc
    • runpy:
      • profile: test_profile
      • trace: test_trace
    • site: test_site

[ ] test: cpython/Lib/test/test_import (TODO: 3)

dependencies:

dependent tests: (no tests depend on import)

[ ] lib: cpython/Lib/test/support
[ ] test: cpython/Lib/test/test_support.py (TODO: 1)
[x] test: cpython/Lib/test/test_script_helper.py

dependencies:

  • support (native: main, _hashlib, _helpers, _hmac, _imp, _interpchannels, _opcode, _remote_debugging, _testcapi, _testinternalcapi, _testlimitedcapi, _thread, _winapi, asyncio.events, collections.abc, concurrent.interpreters, concurrent.interpreters._crossinterp, ctypes.wintypes, email._header_value_parser, errno, faulthandler, gc, hypothesis, hypothesis.configuration, hypothesis.database, import_helper, importlib.machinery, importlib.util, logging.handlers, marshal, math, msvcrt, os.path, os_helper, pwd, resource, script_helper, select, setuptools, setuptools._distutils, sys, time, unicodedata, unittest.case, urllib.error, urllib.parse, urllib.request, zlib)
    • ctypes (native: _ctypes, ctypes._aix, ctypes._endian, ctypes.macholib.dyld, ctypes.macholib.dylib, ctypes.macholib.framework, importlib.machinery, itertools, nt, sys)
    • io (native: _io, _thread, errno, msvcrt, sys)
    • logging (native: atexit, collections.abc, email.message, email.utils, errno, http.client, logging.handlers, multiprocessing.queues, select, sys, time, urllib.parse, win32evtlog, win32evtlogutil)
    • multiprocessing (native: _multiprocessing, _posixshmem, _posixsubprocess, _winapi, array, atexit, collections.abc, connection, context, dummy, errno, forkserver, heap, itertools, managers, mmap, msvcrt, multiprocessing.connection, pool, popen_fork, popen_forkserver, popen_spawn_posix, popen_spawn_win32, queues, resource_sharer, resource_tracker, sharedctypes, spawn, synchronize, sys, time, util, xmlrpc.client)
    • opcode (native: _opcode, builtins)
    • platform (native: _wmi, itertools, java.lang, sys, vms_lib, winreg)
    • sysconfig (native: _sysconfig, _winapi, importlib.machinery, importlib.util, os.path, sys)
    • tkinter (native: _tkinter, itertools, sys, tkinter.commondialog, tkinter.constants, tkinter.dialog, tkinter.simpledialog)
    • unittest (native: _io, _log, async_case, builtins, case, loader, main, os.path, result, runner, signals, suite, sys, time, unittest.util, util)
    • venv (native: _winapi, sys)
    • importlib, warnings
    • _colorize, annotationlib, ast, bz2, codecs, collections, compression, contextlib, dataclasses, datetime, decimal, dis, enum, functools, getopt, getpass, glob, gzip, hashlib, inspect, locale, lzma, os, pathlib, py_compile, re, selectors, shlex, shutil, signal, smtplib, socket, stat, string, struct, subprocess, tempfile, textwrap, threading, tracemalloc, types, zipfile

dependent tests: (2 tests)

  • support: test_pathlib test_pyrepl

[ ] lib: cpython/Lib/importlib
[ ] test: cpython/Lib/test/test_importlib (TODO: 6)

dependencies:

  • importlib

dependent tests: (130 tests)

  • importlib: test_asdl_parser test_bdb test_cmd_line_script test_codecs test_compileall test_ctypes test_doctest test_external_inspection test_frozen test_hashlib test_importlib test_inspect test_linecache test_modulefinder test_multiprocessing_main_handling test_pkgutil test_py_compile test_pyclbr test_pydoc test_pyrepl test_reprlib test_runpy test_sundry test_support test_tomllib test_tools test_unittest test_zipfile test_zipimport test_zoneinfo
    • ctypes.util: test_ctypes
    • ensurepip: test_ensurepip test_venv
    • idlelib: test_idle
    • inspect: test_abc test_argparse test_asyncgen test_buffer test_builtin test_clinic test_code test_collections test_coroutines test_decimal test_enum test_functools test_generators test_grammar test_monitoring test_ntpath test_operator test_patma test_posixpath test_signal test_sqlite3 test_traceback test_turtle test_type_annotations test_type_params test_types test_typing test_unittest test_yield_from test_zipimport_support
      • ast: test_ast test_codeop test_compile test_compiler_codegen test_dis test_fstring test_future_stmt test_peepholer test_peg_generator test_site test_ssl test_type_comments test_ucn test_unparse
      • asyncio: test_asyncio test_concurrent_futures test_logging test_os test_pdb test_unittest
      • cmd: test_cmd
      • dataclasses: test__colorize test_copy test_ctypes test_genericalias test_pprint test_regrtest
      • rlcompleter: test_pyrepl test_rlcompleter
      • trace: test_trace
      • xmlrpc.server: test_docxmlrpc test_xmlrpc
    • profile: test_profile
    • py_compile: test_importlib
      • zipfile: test_shutil test_zipapp test_zipfile test_zipfile64
    • sysconfig: test_c_locale_coercion test_cmd_line test_dtrace test_embed test_gc test_importlib test_launcher test_osx_env test_peg_generator test_posix test_pyexpat test_subprocess test_sys test_sysconfig test_time test_tools test_urllib2net
    • zipfile:
      • shutil: test_bz2 test_filecmp test_glob test_httpservers test_largefile test_sax test_string_literals test_tarfile test_tempfile test_unicode_file
    • zipimport: test_importlib

Legend:

  • [+] path exists in CPython
  • [x] up-to-date, [ ] outdated

@fanninpm
fanninpm requested a review from youknowone October 2, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@youknowone youknowone Awaiting requested review from youknowone

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@youknowdot