Skip to content

[subinterpreters] Static types incorrectly share some objects between interpreters #94673

Closed
Closed
[subinterpreters] Static types incorrectly share some objects between interpreters#94673
Assignees
ericsnowcurrently
Labels
3.12only security fixestopic-subinterpreterstype-featureA feature request or enhancement

Description

@ericsnowcurrently
ericsnowcurrently
opened on Jul 7, 2022

While static types (PyTypeObject values) don't themselves ever change (once PyType_Ready() has run), they do hold mutable data. This means they cannot be safely shared between multiple interpreters without a common GIL (and without state leaking between).

Mutable data:

  • the object header (e.g. refcount), as with all objects
  • otherwise immutable objects:
    • ob_type (__class__)
    • tp_base (__base__) - set by PyType_Ready() if not set
    • tp_bases (__bases__) - always set by PyType_Ready()
    • tp_mro (__mro__) - always set by PyType_Ready()
    • tp_dict (__dict__) - set by PyType_Ready() if not set
  • mutable containers:
    • tp_subclasses (__subclasses__)
    • tp_weaklist

(See https://docs.python.org/3/c-api/typeobj.html#tp-slots.)
(Note that tp_cache is no longer used.)

For the object header, if PEP 683 (immortal objects) is accepted then we can make static types immortal.

For the otherwise immutable objects, we can make sure each is immortal and then we're good. Even tp_dict is fine since it gets hidden behind types.MappingProxyType. We'd also need to either make sure each contained item is immortal.

For tp_subclasses we will need a per-interpreter copy, and do the proper lookup in the __subclasses__ getter. The cache could be stored on PyInterpreterState or even as a dict in tp_subclasses.

For tp_weaklist it's a similar story as for tp_subclasses. Note that tp_weaklist isn't very important for static types since they are never deallocated.

(The above is also discussed in PEP 684.)

CC @kumaraditya303

Linked PRs

Activity

  1. ericsnowcurrently
    added
    type-featureA feature request or enhancement
    on Jul 7, 2022
  2. ericsnowcurrently
    self-assigned this
    on Jul 7, 2022
  3. ericsnowcurrently
    added this to Subinterpreterson Jul 7, 2022
  4. ericsnowcurrently
    moved this to In Progress in Fancy CPython Boardon Jul 7, 2022
  5. ericsnowcurrently
    added this to Fancy CPython Boardon Jul 7, 2022
  6. AlexWaygood
    added
    topic-subinterpreters
    on Jul 8, 2022

  7. ericsnowcurrently commented on Jul 9, 2022

    @ericsnowcurrently
    ericsnowcurrently
    on Jul 9, 2022
    MemberAuthor

    See https://peps.python.org/pep-0684/#global-objects.

  8. kumaraditya303 commented on Jul 10, 2022

    @kumaraditya303
    kumaraditya303
    on Jul 10, 2022
    Contributor

    The current _PyInterpreterState_GET (which would be used here) static inline function is quite inefficient as there is a lot indirection and involves atomic reads to get the "current" thread and then get the interpreter state. To improve this we can use thread local 1 variables on supported platform such as Linux, Windows and macOS. On Linux with glibc, the thread local implementation uses fs segment register and on x86-64 it is very efficient compared to the current implementation.
    This isn't required here but would be nice to investigate separately.

    Footnotes

    1. https://en.cppreference.com/w/c/thread/thread_local ↩

  9. encukou commented on Jul 11, 2022

    @encukou
    encukou
    on Jul 11, 2022
    Member

    This means they cannot be safely shared between multiple interpreters without a common GIL

    Python currently has a common GIL.
    Shouldn't this be part of a PEP, rather than an issue?

  10. ericsnowcurrently
    added a commit that references this issue on Jul 19, 2022

    gh-94673: [c-analyzer] Add a Script to Identify Static Types (#94989)

    7a1a85d
  11. ericsnowcurrently
    moved this from In Progress to In Review in Fancy CPython Boardon Jul 19, 2022
  12. ericsnowcurrently
    added
    3.12only security fixes
    on Jul 19, 2022
  13. ericsnowcurrently
    moved this to In Progress in Subinterpreterson Jul 19, 2022

  14. ericsnowcurrently commented on Jul 19, 2022

    @ericsnowcurrently
    ericsnowcurrently
    on Jul 19, 2022
    MemberAuthor

    I have the changes ready for this:

  15. 28 remaining items

  16. carljm
    added 2 commits that reference this issue on May 2, 2023

    Merge branch 'main' into inlinecomp2

    0727d6f

    Merge branch 'main' into cached_property_docs

    b9c39e3
  17. ericsnowcurrently
    added a commit that references this issue on May 3, 2023

    gh-94673: More Per-Interpreter Fields for Builtin Static Types (gh-10…

    de64e75
  18. ericsnowcurrently
    added a commit that references this issue on Jun 7, 2023

    gh-94673: Ensure subtypes are readied only once in math.trunc() (gh-1…

    5394bf9
  19. miss-islington
    added a commit that references this issue on Jun 7, 2023

    pythongh-94673: Ensure subtypes are readied only once in math.trunc() (…

    7a715c2
  20. ericsnowcurrently
    added a commit that references this issue on Jun 7, 2023

    [3.12] gh-94673: Ensure subtypes are readied only once in math.trunc() (

    1e12877
  21. ericsnowcurrently
    added a commit that references this issue on Apr 12, 2024

    gh-94673: Clarify About Runtime State Related to Static Builtin Types (…

    eca5362

  22. gvanrossum commented on Apr 15, 2024

    @gvanrossum
    gvanrossum
    on Apr 15, 2024
    Member

    We now get a warning when building without pydebug:

    Objects/typeobject.c:120:1: warning: unused function 'static_builtin_index_is_set' [-Wunused-function]
static_builtin_index_is_set(PyTypeObject *self)
^
1 warning generated.

  23. ericsnowcurrently commented on Apr 16, 2024

    @ericsnowcurrently
    ericsnowcurrently
    on Apr 16, 2024
    MemberAuthor

    I'll take a look.

  24. diegorusso
    added a commit that references this issue on Apr 17, 2024

    pythongh-94673: Clarify About Runtime State Related to Static Builtin…

    8d71390
  25. erlend-aasland
    added 2 commits that reference this issue on Apr 17, 2024

    pythongh-94673: Fix compiler warning in typeobject.c

    9ad94d3

    gh-94673: Fix compiler warning in typeobject.c (#117980)

    353ea0b
  26. navytux
    added a commit that references this issue on Apr 4, 2025

    golang_str: Fix SIGSEGV on import on py3.12

    4524dfc
  27. navytux
    added a commit that references this issue on Mar 18, 2026

    golang_str: Fix infrastructure to patch builtin types for py3.13+

    a614079
  28. zhuyifei1999
    added a commit that references this issue on Apr 7, 2026

    heapyc & setsc: Lock static type initializations

    75476b4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.12only security fixestopic-subinterpreterstype-featureA feature request or enhancement

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions