Skip to content

Match remaining native docs and signatures to CPython 3.14 - #8877

Merged
youknowone merged 7 commits into
RustPython:mainfrom
youknowone:whats-left-signatures
Sep 29, 2026
Merged

youknowone merged 7 commits into
RustPython:mainfrom
youknowone:whats-left-signatures

Conversation

@youknowone

@youknowone youknowone commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #8836. Fixes the remaining native __doc__/signature mismatches that scripts/whats_left.py --doc --signature reports under CPython 3.14.7, except those from modules RustPython implements in pure Python.

Doc DB generator (crates/doc/generate.py)

  • __dir__ was in IGNORED_ATTRS, so object.__dir__, type.__dir__ etc. were never recorded. It is no longer ignored; __base__, __bases__ and __module__ are skipped instead because following them only re-records other objects under a bogus path (this removes 22 builtins.type.__base__* keys).
  • New keys that the RustPython lookup needs:
    • native classes whose metaclass is not type, and classes stored under another attribute name keyed by their own __name__ (_json.make_scanner is the class _json.Scanner)
    • native objects exposed by pure-Python modules, keyed by their own __module__/__name__ (ssl.SSLError), or by the exposing module when __module__ is None (the codecs.*_errors handlers)
    • types found through object.__subclasses__() keyed by their own module as well as builtins (_ctypes._CData)
    • getset/member descriptors named like class metadata, which getmembers() hides behind the class's own value (coroutine.__name__)
    • native types a module exposes only through an instance (sys.flags, sys.float_info, …)
  • A class counts as native only when it owns its C descriptors (__objclass__ is cls). Before, an enum holding a copied str.__format__ looked native.
  • Docstrings are stored unstripped. .strip() removed leading indentation that inspect.cleandoc depends on (_sre.template).
  • Output is deterministic: all modules are imported, in sorted order, before any class is classified. Before, the result depended on import order and PYTHONHASHSEED.

data.inc.rs gets the key delta between the old and new generator, both run on macOS with CPython 3.14.7: 5,042 added, 22 removed, 1,773 values changed (mostly unstripped whitespace). Linux/Windows-only keys are picked up by the next update-doc-db run.

/// on Python items

Rust /// comments on #[py*] items were published as __doc__ (e.g. weakref.ref was "PyWeakReference: …"). They are now //. Most of this is split out as #8893; this PR keeps the struct sequence __replace__ and coroutine.__name__/__qualname__ ones, which need the database entries added here.

Names, lookup and signatures

  • The weakref type is weakref.ReferenceType; _ctypes.Array.__class_getitem__ is defined on Array, not its metaclass.
  • #[pymodule(sub)] bodies declare the Python module they are merged into with name = "...", and doc lookup uses it; without name the database is not consulted. This replaces a guess that searched every module for a unique function name. posix and nt fall back to each other in rustpython_doc::get_qualified, since they are one module under a platform-dependent name. Each name of a multi-name function gets its own doc (posix.remove / posix.unlink).
  • As a result posix.DirEntry/ScandirIter have __module__ posix (was _os), and odict_keys/odict_values/odict_items/odict_iterator have builtins (was ordered_dict).
  • Methods coming from with(...) traits get the implementing class's doc (dict_items.__reversed__, dict_items.isdisjoint, dict_values.__reversed__). The trait impl cannot name the class, so the derive marks such a method doc_body_pending on its PyMethodDef, and the class fills the span from its attribute doc table when it concatenates the method arrays.
  • #[pymodule] classes without their own doc get the database doc, resolved at macro time. The codec error handlers and _lzma.LZMAError resolve theirs in const context through the new ItemDoc::db. Context::new_exception_type_with_doc is added for the latter. No database lookup happens at runtime, so the key table stays out of the binary.
  • posix_spawn/posix_spawnp: parameter argv (the field is renamed rather than overridden with name =) and defaults file_actions=(), setpgroup=None, setsigmask=(), setsigdef=(), scheduler=None in CPython's order.
  • dict_keys and dict_items are unhashable; _CData.__ctypes_from_outparam__ is added; sys.__doc__ omits the [Windows only] lines on other platforms.

Struct sequence fields

Since #8857 made struct sequence fields member descriptors, their __doc__ has been None (os.stat_result.st_mode, time.struct_time.tm_year, _thread._ExceptHookArgs.exc_type). new_readonly_tuple_member now takes the field's database doc.

Results (macOS arm64, CPython 3.14.7 as reference)

whats_left before after
__doc__ mismatches outside decimal/datetime/zoneinfo 45 0
signature mismatches outside decimal/datetime/zoneinfo and environment noise¹ 2 0
builtin items (dict_items.__hash__) 1 0

A __doc__ dump over every builtin module's classes, functions and descriptors (5,168 entries) matches CPython in 201 more places than 1a259e4 and in none fewer.

The remaining 267 doc mismatches are all decimal.*, datetime.* and zoneinfo.*: RustPython uses _pydecimal, _pydatetime and _zoneinfo's pure-Python fallbacks there.

¹ functools._make_key, logging.shutdown, platform.architecture and _py_warnings._deprecated render set ordering, paths or sys.version_info in their defaults.

Release binary: 38,088,784 bytes, against 38,048,448 for a 1a259e4 build (+40 KB, mostly the new struct sequence field and attribute doc tables).

Verified with workspace + capi clippy (-D warnings), cargo check --no-default-features without doc, and cargo test --workspace. These unittests pass: test_inspect, test_pydoc, test_weakref, test_ctypes, test_json, test_codecs, test_ssl, test_posix, test_os, test_dict, test_sys, test_lzma, test_builtin, test_descr, test_types, test_zlib, test_csv, test_pickle, test_coroutines, test_time, test_structseq, test_thread and test_threading. extra_tests shows no new failures.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Important

Review skipped

We couldn't safely recover the incremental review. No full review was started, and the last reviewed checkpoint was preserved. Retry later, or explicitly request a full review by commenting @coderabbitai full review.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: RustPython/RustPython/.coderabbit.yml

Review profile: CHILL

Plan: Advanced

Run ID: 87475a4a-bdf9-41a0-9e0c-a0d4f4d3ca8b

📥 Commits

Reviewing files that changed from the base of the PR and between 908368e and 93ca4a4.

📒 Files selected for processing (44)
  • crates/derive-impl/src/pyclass.rs
  • crates/derive-impl/src/pymodule.rs
  • crates/doc/generate.py
  • crates/doc/src/data.inc.rs
  • crates/stdlib/src/_asyncio.rs
  • crates/stdlib/src/cjkcodecs/multibytecodec.rs
  • crates/stdlib/src/elementtree.rs
  • crates/stdlib/src/json.rs
  • crates/stdlib/src/lzma.rs
  • crates/stdlib/src/multiprocessing.rs
  • crates/stdlib/src/pickle.rs
  • crates/stdlib/src/resource.rs
  • crates/stdlib/src/ssl.rs
  • crates/vm/src/builtins/asyncgenerator.rs
  • crates/vm/src/builtins/coroutine.rs
  • crates/vm/src/builtins/descriptor.rs
  • crates/vm/src/builtins/dict.rs
  • crates/vm/src/builtins/object.rs
  • crates/vm/src/class.rs
  • crates/vm/src/codecs.rs
  • crates/vm/src/exceptions.rs
  • crates/vm/src/frame.rs
  • crates/vm/src/function/doctext.rs
  • crates/vm/src/function/method.rs
  • crates/vm/src/object/core.rs
  • crates/vm/src/stdlib/_abc.rs
  • crates/vm/src/stdlib/_ctypes.rs
  • crates/vm/src/stdlib/_ctypes/array.rs
  • crates/vm/src/stdlib/_ctypes/base.rs
  • crates/vm/src/stdlib/_ctypes/function.rs
  • crates/vm/src/stdlib/_ctypes/pointer.rs
  • crates/vm/src/stdlib/_ctypes/simple.rs
  • crates/vm/src/stdlib/_ctypes/structure.rs
  • crates/vm/src/stdlib/_ctypes/union.rs
  • crates/vm/src/stdlib/_interpchannels.rs
  • crates/vm/src/stdlib/_io.rs
  • crates/vm/src/stdlib/_thread.rs
  • crates/vm/src/stdlib/_winapi.rs
  • crates/vm/src/stdlib/gc.rs
  • crates/vm/src/stdlib/posix.rs
  • crates/vm/src/stdlib/sys.rs
  • crates/vm/src/types/structseq.rs
  • crates/vm/src/vm/context.rs
  • crates/wasm/src/js_module.rs

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

The changes expand documentation discovery and runtime documentation assignment. They also update several Python-visible API declarations, add a ctypes method, and adjust source comments and module documentation.

Changes

Documentation discovery and exposure

Layer / File(s) Summary
Discover documentation entries from standard-library modules
crates/doc/generate.py
The generator traverses native objects exposed by Python modules, records selected builtins under eligible module names, and adjusts docstring processing and traversal.
Look up and assign runtime documentation
crates/vm/src/function/*, crates/vm/src/class.rs, crates/vm/src/vm/context.rs, crates/vm/src/codecs.rs, crates/derive-impl/src/*, crates/stdlib/src/lzma.rs, crates/vm/src/stdlib/sys.rs, crates/vm/src/stdlib/_thread.rs
Method definitions can use attribute documentation. Generated classes, exception types and selected handlers receive documentation. Submodule function lookup adds fallbacks for shared names, and sys.__doc__ is filtered on non-Windows platforms.
Adjust source documentation annotations
crates/stdlib/src/*, crates/vm/src/builtins/*, crates/vm/src/stdlib/*, crates/vm/src/types/structseq.rs, crates/vm/src/frame.rs, crates/vm/src/exceptions.rs, crates/wasm/src/js_module.rs
Many Rust documentation comments become ordinary comments. Some comment wording also changes. These edits do not change the described executable implementations.

Python-visible API and metadata updates

Layer / File(s) Summary
Update Python-visible declarations
crates/vm/src/builtins/dict.rs, crates/vm/src/object/core.rs, crates/vm/src/stdlib/posix.rs
Dictionary view classes receive explicit hashability settings. PyWeak is exposed as weakref.ReferenceType. PosixSpawnArgs changes its argument name, defaults and declaration order.
Add ctypes out-parameter method
crates/vm/src/stdlib/_ctypes/base.rs
PyCData gains __ctypes_from_outparam__, which returns the same object as a PyObjectRef.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 93ca4

No identified compatibility issue remains that should prevent merging after normal checks.

Architecture Summary

Architecture risk: 🔵 Low · up to 93ca4

The change affects 1 system.

Changed systems: crates

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — crates (service) was modified; 43 changed files map to changed impact.

Before / after behavior

  • observed — Modified behavior in crates/derive-impl/src/pyclass.rs: The generated METHOD_DEFS expression now uses concat_with_attr_docs and the payload type’s ATTR_DOCS when compiled with doc; without doc, it continues to use __const_concat_arrays. Both paths concatenate the own and included method-definition arrays with the same computed length.
  • observed — Modified behavior in crates/derive-impl/src/pymodule.rs: Adds submodule_func_doc, which uses a unique-owner doc when available and otherwise searches nonempty posix and nt docs; updates the accompanying documentation comment to describe shared ownership.
  • observed — Modified behavior in crates/derive-impl/src/pymodule.rs: Submodule function doc lookup now uses submodule_func_doc instead of unique_submodule_func_doc, allowing the added posix/nt fallback when unique lookup yields no doc.
  • observed — Modified behavior in crates/derive-impl/src/pymodule.rs: Class generation now searches for a nonempty doc under the class’s own module first, then its publication module, and prepares a call to assign the found doc to the new class.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 80.62% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 129 functions across 42 files. (1 skipped: …
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main changes: aligning remaining native documentation and signatures with CPython 3.14.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@youknowone
youknowone force-pushed the whats-left-signatures branch from 93ca4a4 to 2c3ca84 Compare September 28, 2026 10:54
@codspeed

codspeed Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will improve performance by 10.54%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 61 untouched benchmarks
⏩ 4 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ rustpython[richards.py] 1.9 ms 1.7 ms +10.54%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing youknowone:whats-left-signatures (c5540e9) with main (b499e70)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

- Stop ignoring `__dir__`; skip `__base__`, `__bases__` and `__module__` instead.
- Recurse into native classes whose metaclass is not `type`, and also key a
  class stored under another attribute name by its own `__name__`.
- Collect native objects exposed by pure-Python modules, such as the codec
  error handlers, under their own `__module__` and `__name__`.
- Key types found through `object.__subclasses__()` by their own module as
  well as `builtins`.
- Record getset/member descriptors named like class metadata
  (`coroutine.__name__`), and native types a module exposes only through an
  instance (`sys.flags`).
- Treat a class as native only when it owns its C descriptors, so an enum
  holding a copied `str.__format__` is skipped.
- Keep docstrings unstripped, and import every module in sorted order before
  walking them.
- Apply the resulting key delta, measured on macOS with CPython 3.14.7, to
  data.inc.rs.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
Use `//` instead of `///` on Python-exposed items so their comments are not
published as `__doc__`.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
- Rename the weakref type to `weakref.ReferenceType` and define
  `__class_getitem__` on `_ctypes.Array` instead of its metaclass.
- Give each name of a multi-name `#[pymodule(sub)]` function
  (`posix.remove`/`unlink`) its database doc.
- Fill methods from `with(...)` traits (dict view `__reversed__`,
  `isdisjoint`) with the implementing class's database doc.
- Resolve database docs for `#[pymodule]` classes at macro time, and for the
  codec error handlers and `_lzma.LZMAError` in const context.
- Add `Context::new_exception_type_with_doc`.
- Match `posix_spawn` parameter names and defaults, make `dict_keys` and
  `dict_items` unhashable, add `_CData.__ctypes_from_outparam__`, and omit
  the Windows-only lines of `sys.__doc__` on other platforms.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
`new_readonly_tuple_member` takes the field doc, looked up in the class's
attribute doc table through the new `class_attr_item_doc`.

Assisted-by: Claude:claude-opus-5-5
posix and nt share one doc entry. A sub module sets name to the
Python module it is merged into, and doc lookup uses that name.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
The pymethod derive sets doc_body_pending when a method has no Rust
doc body and expansion did not resolve a database span.
concat_with_attr_docs copies that span from the class table and
clears the flag. Other constructors leave the flag false.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Grok:grok-4.7
PosixSpawnArgs.args is renamed to argv, and nt's follow_symlinks field
drops a name override equal to the field name.

Assisted-by: Claude:claude-opus-5-5
@youknowone
youknowone force-pushed the whats-left-signatures branch from 19f158f to c5540e9 Compare September 28, 2026 17:20
@youknowone
youknowone merged commit 04eb0ad into RustPython:main Sep 29, 2026
30 checks passed
@youknowone
youknowone deleted the whats-left-signatures branch September 29, 2026 02:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant