Skip to content

[subinterpreters] Meta issue: per-interpreter GIL #84692

Closed
Closed
[subinterpreters] Meta issue: per-interpreter GIL#84692
Labels
3.11only security fixestopic-subinterpreterstype-featureA feature request or enhancement

Description

@vstinner
vstinner
opened on May 5, 2020
BPO 40512
Nosy @vstinner, @jparise, @encukou, @markshannon, @ericsnowcurrently, @ndjensen, @corona10, @shihai1991, @aeros, @erlend-aasland, @nw0
PRs
  • bpo-40512: Store pointer to interpreter state in a thread local variable #29228
    		•
    Files
  • demo-pyperf.py
  • resolve_slotdups.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2020-05-05.12:51:15.115>
labels = ['expert-subinterpreters', 'type-feature', '3.11']
title = '[subinterpreters] Meta issue: per-interpreter GIL'
updated_at = <Date 2022-03-06.03:51:12.908>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2022-03-06.03:51:12.908>
actor = 'jon'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Subinterpreters']
creation = <Date 2020-05-05.12:51:15.115>
creator = 'vstinner'
dependencies = []
files = ['49128', '49700']
hgrepos = []
issue_num = 40512
keywords = ['patch']
message_count = 30.0
messages = ['368136', '368138', '368142', '368184', '368195', '368203', '368206', '368210', '368272', '368310', '368670', '368839', '368908', '368914', '370608', '372253', '372773', '380102', '380108', '380323', '383780', '383830', '383831', '383875', '388746', '399847', '401117', '401127', '401130', '403727']
nosy_count = 12.0
nosy_names = ['vstinner', 'jon', 'petr.viktorin', 'Mark.Shannon', 'eric.snow', 'ndjensen', 'corona10', 'alex-garel', 'shihai1991', 'aeros', 'erlendaasland', 'nw0']
pr_nums = ['29228']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40512'
versions = ['Python 3.11']

    Activity

    1. vstinner commented on May 5, 2020

      @vstinner
      vstinner
      on May 5, 2020
      MemberAuthor

      To be able to run multiple (sub)interpreters in parallel, the unique global interpreter lock aka "GIL" should be replace with multiple GILs: one "GIL" per interpreter. The scope of such per-interpreter GIL would be a single interpreter.

      The current CPython code base is not fully read to have one GIL per interpreter. TODO:

      Until we can ensure that no Python object is shared between two interpreters, we might make PyObject.ob_refcnt, PyGC_Head (_gc_next and _gc_prev) and _dictkeysobject.dk_refcnt atomic.

      C extension modules should be modified as well:

      • Convert to PEP-489 multi-phase initialization
      • Replace globals ("static" variables) with a module state, or design a new "per-interpreter" local storage similar to Thread Local Storage (TLS). There is already PyInterpreterState.dict which is convenient to use in "Python" code, but it's not convenient to use in "C" code (code currently using "static int ..." for example).

      I'm not sure how to handle C extensions which are binding for a C library which has a state and so should not be used multiple times in parallel. Some C extensions use a "global lock" for that. The question is how to get

      Most of these tasks are already tracked in Eric Snow's "Multi Core Python" project:
      https://github.com/ericsnowcurrently/multi-core-python/issues

      This issue is related to PEP-554 "Multiple Interpreters in the Stdlib", but not required by this PEP.

      This issue is a tracker for sub-issues related to the goal "have one GIL per interpreter".

      --

      Some changes have a negative impact on "single threaded" Python application. Even if the overhead is low, one option to be able to move faster on this issue may be to add a new temporary configure option to have an opt-in build mode to better isolate subinterpreters. Examples:

      • disable pymalloc
      • atomic reference counters
      • disable free lists

      That would be a temporary solution to "unblock" the development on this list. For the long term, free lists should be made per-interpreter, pymalloc should support multiple interpreters, no Python object must be shared by two interpreters, etc.

      --

      One idea to detect if a Python object is shared by two interpreters *in debug mode* would be to store a reference to the interpreter which created it, and then check if the current interpreter is the same. If not, fail with a Python Fatal Error.

      --

      During Python 3.9 development cycle, many states moved from the global _PyRuntimeState to per-interpreter PyInterpreterState:

      Many corner cases related to daemon threads have also been fixed:

      And more code is now shared for the initialization and finalization of the main interpreter and subinterpreters (ex: see bpo-38858).

      Subinterpreters builtins and sys are now really isolated from the main interpreter (bpo-38858).

      --

      Obviously, there are likely tons of other issues which are not known at this stage. Again, this issue is a placeholder to track them all. It may be more efficient to create one sub-issue per sub-task, rather than discussing all tasks at the same place.

    2. vstinner
      added
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      3.9 (EOL)end of life
      type-featureA feature request or enhancement
      on May 5, 2020

    3. vstinner commented on May 5, 2020

      @vstinner
      vstinner
      on May 5, 2020
      MemberAuthor

      Move signals pending and gil_drop_request from _PyRuntimeState.ceval to PyInterpreterState.ceval: ericsnowcurrently/multi-core-python#34

      I created bpo-40513: "Move _PyRuntimeState.ceval to PyInterpreterState".

    4. vstinner commented on May 5, 2020

      @vstinner
      vstinner
      on May 5, 2020
      MemberAuthor

      Some changes have a negative impact on "single threaded" Python application. Even if the overhead is low, one option to be able to move faster on this issue may be to add a new temporary configure option to have an opt-in build mode to better isolate subinterpreters. (...)

      I created bpo-40514: "Add --experimental-isolated-subinterpreters build option".

    5. vstinner commented on May 5, 2020

      @vstinner
      vstinner
      on May 5, 2020
      MemberAuthor

      I created bpo-40522: "Subinterpreters: get the current Python interpreter state from Thread Local Storage (autoTSSkey)".

    6. vstinner commented on May 5, 2020

      @vstinner
      vstinner
      on May 5, 2020
      MemberAuthor

      Attached demo.py: benchmark to compare performance of sequential execution, threads and subinterpreters.

    7. vstinner commented on May 5, 2020

      @vstinner
      vstinner
      on May 5, 2020
      MemberAuthor

      (oops, there was a typo in my script: threads and subinterpreters was the same benchmark)

    8. vstinner commented on May 5, 2020

      @vstinner
      vstinner
      on May 5, 2020
      MemberAuthor

      Hum, demo.py is not reliable for threads: the standard deviation is quite large. I rewrote it using pyperf to compute the average and the standard deviation.

    9. vstinner commented on May 5, 2020

      @vstinner
      vstinner
      on May 5, 2020
      MemberAuthor

      I updated demo-pyperf.py to also benchmark multiprocessing.

    10. vstinner commented on May 6, 2020

      @vstinner
      vstinner
      on May 6, 2020
      MemberAuthor

      I created bpo-40533: "Subinterpreters: don't share Python objects between interpreters".

    11. 26 remaining items

    12. vstinner commented on Aug 18, 2021

      @vstinner
      vstinner
      on Aug 18, 2021
      MemberAuthor

      PyStructSequence_InitType2() is not compatible with subinterpreters: it uses static types. Moreover, it allocates tp_members memory which is not released when the type is destroyed. But I'm not sure that the type is ever destroyed, since this API is designed for static types.

    13. shihai1991 commented on Sep 6, 2021

      @shihai1991
      shihai1991
      on Sep 6, 2021
      Member

      PyStructSequence_InitType2() is not compatible with subinterpreters: it uses static types. Moreover, it allocates tp_members memory which is not released when the type is destroyed. But I'm not sure that the type is ever destroyed, since this API is designed for static types.

      IMO, I suggest to create a new function, PyStructSequence_FromModuleAndDesc(module, desc, flags) to create a heaptype and don't aloocates memory block for tp_members,something like 'PyType_FromModuleAndSpec()`.

      I don't know there have any block issue to do this converting operation. But I can take a look.

      @petr ping, Petr, do you have any better idea about this question :)

    14. vstinner commented on Sep 6, 2021

      @vstinner
      vstinner
      on Sep 6, 2021
      MemberAuthor

      Hai Shi:

      IMO, I suggest to create a new function, PyStructSequence_FromModuleAndDesc()

      Please create a new issue. If possible, I would prefer to have a sub-issue for that, to keep this issue as a tracking issue for all issues related to subinterpreters.

    15. shihai1991 commented on Sep 6, 2021

      @shihai1991
      shihai1991
      on Sep 6, 2021
      Member

      bpo-45113: [subinterpreters][C API] Add a new function to create PyStructSequence from Heap.

    16. shihai1991
      added
      3.11only security fixes
      and removed
      3.10 (EOL)end of life
      on Sep 6, 2021

    17. encukou commented on Oct 12, 2021

      @encukou
      encukou
      on Oct 12, 2021
      Member

      PyStructSequence_NewType exists, and is the same as the proposed PyStructSequence_FromModuleAndDesc except it doesn't take the module (which isn't necessary: PyStructSequence_Desc has no way to define functionality that would need the module state).

    18. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    19. ezio-melotti
      added this to Subinterpreterson Apr 15, 2022
    20. ezio-melotti
      moved this to Todo in Subinterpreterson Apr 15, 2022

    21. vstinner commented on Nov 3, 2022

      @vstinner
      vstinner
      on Nov 3, 2022
      MemberAuthor

      Sadly, I don't have the bandwidth to work on this issue, so I just close it.

      @ericsnowcurrently is now working on https://peps.python.org/pep-0684/ which is a little bit different.

    22. vstinner
      closed this as completedon Nov 3, 2022
    23. Repository owner moved this from Todo to Done in Subinterpreterson Nov 3, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.11only security fixestopic-subinterpreterstype-featureA feature request or enhancement

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions