Split InterpreterFrame from FrameObject for stack-allocated execution - #8354

Summary

Split InterpreterFrame (execution state) from FrameObject (Python-visible frame object) so that normal function calls stack-allocate only an InterpreterFrame on the Rust stack. A full FrameObject is created lazily via materialize() only when Python code observes the frame (e.g. sys._getframe(), traceback creation, sys.settrace()).

This mirrors CPython 3.11+ where _PyInterpreterFrame lives on the C stack and PyFrameObject is allocated on demand.

Architecture

• InterpreterFrame — #[repr(C)] struct on the Rust stack inside with_iframe. Contains code, globals, builtins, localsplus, lasti, trace state, and a previous pointer forming the TLS frame chain.
• FrameObject — PyObject wrapper created by materialize() when needed. Holds an owned copy of InterpreterFrame and is linked via materialized / find_live_source_iframe() for bidirectional access.
• with_iframe — new fast path for regular function calls. No heap allocation, no refcount, no freelist.
• with_frame — existing path for generators, coroutines, exec(), eval() that require a durable FrameObject.

Key design decisions

current_code() and free-threading safety: current_code() reads the topmost InterpreterFrame from thread-local CURRENT_FRAME and borrows its code pointer. This is safe because: (1) CURRENT_FRAME is per-thread TLS — no cross-thread access; (2) the code pointer borrows from the PyFunction on the caller's stack, which is alive while the frame executes; (3) .to_owned() increments the refcount before returning, producing an independent PyRef<PyCode>.

Cross-thread frame access: f_back, sys._current_frames(), and sys._current_exceptions() use stop-the-world (STW) to safely materialize cross-thread iframe chains. STW is entered before dereferencing any cross-thread pointer to prevent use-after-free races. The non-unix path now uses STW identically to unix, replacing the previous frames Mutex fallback.

set_f_lineno (debugger jump): Writes lasti, pending_stack_pops, and pending_unwind_from_stack to the live source iframe via find_live_source_iframe(), not the materialized copy, so pdb jump commands take effect on stack-allocated frames.

GC tracking timing: Materialized FrameObjects are tracked in GC only at with_iframe cleanup after set_current_frame restores the old chain. This prevents premature collection while the frame is still executing.

retained_back: Only captures already-materialized callers to avoid adding refcounts on local variables (which would delay __del__ / ResourceWarning). For non-materialized callers, f_back resolves via the TLS chain while executing, or returns None after return.

Performance improvements

• Zero heap allocation for normal function calls (no FrameObject, no freelist, no refcount)

• Amortized C stack overflow check (every 8th recursion depth)

• Panic-safe recursion depth via scopeguard in with_frame

Benchmark results (Apple M-series, release build)

Test fixes included

• test_sys (Windows): full frame chain materialization with retained_back in non-unix get_all_current_frames
• test_current_exceptions (Windows): STW-based cross-thread f_back on all platforms
• test_frame: frame.clear() rejects live frames, f_locals proxy writes to live iframe
• test_traceback: deferred GC tracking for materialized frames
• test_generators: removed @expectedFailure for frame/GC cycle tests
• test_pdb: f_trace propagation to live source iframe
• test_faulthandler: use top_iframe for all frame chain walking

Addresses youknowone#40

Summary by CodeRabbit

• Performance

  • Improved execution efficiency with a lightweight fast-frame path for eligible code.
  • Reduced overhead from recursion and stack checks through periodic native stack probing.

• Bug Fixes

  • Improved traceback formatting, frame navigation, and depth reporting across threads and lightweight frames.
  • Improved tracing, profiling, warnings, imports, built-in introspection, and sys._getframe behavior.
  • Tightened text encoding validation for consistent NUL detection.
  • Improved frame and locals visibility for generators, coroutines, and asynchronous generators.