Repository navigation
Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable #80891
Description
Activity
Eric Snow moved global variables into a _PyRuntimeState structure which is made of sub-structures. There is a single instance of _PyRuntimeState: the _PyRuntime global variable.
I would like to add "_PyRuntimeState *" parameters to functions to avoid relying directly on _PyRuntime global variable. The long term goal is to have "stateless" code: don't rely on global variables, only on input parameters. In practice, we will continue to use thread local storage (TLS) to get the "current context" like the current interpreter and the current Python thread state.
I don't really understand the rationale for these changes. What's wrong with the global variable _PyRuntime?
What's the long-term goal for _PyRuntime? If you can't get rid of all occurrences of _PyRuntime, how does it help to get rid of *some* occurences?
I don't really understand the rationale for these changes. What's wrong with the global variable _PyRuntime?
What's the long-term goal for _PyRuntime? If you can't get rid of all occurrences of _PyRuntime, how does it help to get rid of *some* occurences?The long term goal is to support multiple interpreter instances per process:
Eric Snow's PEP-554 "Multiple Interpreters in the Stdlib"
https://www.python.org/dev/peps/pep-0554/Right now, there is a single instance of _PyRuntimeState: _PyRuntime. I would be interested to have one _PyRuntimeState per interpreter.
Maybe _PyRuntimeState should be reworked in the meanwhile. Right now, it's a strange beast: it contains things which are set before Python initialization and things which are set after. It contains C types and Python objects. Maybe some parts should be moved into PyInterpreterState or even PyThreadState. I don't know at this point. It takes time to look at each individual structure field...
Anyway, more generally, IMHO it's a bad practice to rely on a global variable. Python runtime should be "stateless".
The current implementation of CPython leaks dozens of *Python* objects at exit. For example, I started to work on this issue while working on https://bugzilla.redhat.com/show_bug.cgi?id=1696322 : Python doesn't clear 2 warnings variables at exit. When I looked into this issue, I also noticed that _PyRuntime.gc.garbage remains *alive* after Py_Finalize().
That's plain wrong: *all* Python objects must be cleared by Py_Finalize(). Two interpreters must *not* share any Python object.
Well, the PEP should better explain the rationale than me :-)
--
When I wrote PR 12934, I noticed that even getting the current thread state rely on _PyRuntime:
#define _PyThreadState_GET() \ ((PyThreadState*)_Py_atomic_load_relaxed(&_PyRuntime.gilstate.tstate_current))
That's wrong in the case of ceval.c: it should be possible to run _PyEval_EvalFrameDefault() twice at the same time in two threads using two "isolated" interpreters.
Well, PR 12934 doesn't fix all issues. It's just one small step towards "stateless" runtime and the even more long term of having one GIL per *interpeter*, rather than a single GIL per *process*.
The long term goal is to support multiple interpreter instances per process:
Eric Snow's PEP-554 "Multiple Interpreters in the Stdlib"
https://www.python.org/dev/peps/pep-0554/Sorry, but I don't see the relation between this issue and PEP-554. It seems to me that the PEP is about making subinterpreters available from pure Python (instead of only at the C level). It doesn't say anything about the *implementation* of subinterpreters, which is what this issue is about.
So I'm still missing the bigger picture where this issue fits in.
The current implementation of CPython leaks dozens of *Python* objects at exit.
That may be an issue to be fixed, but again I don't see the relation with this issue.
Jeroen Demeyer:
Sorry, but I don't see the relation between this issue and PEP-554.
The long term plan for PEP-554 is to support having one GIL per interpreter for best performances. The GIL lives in _PyRuntime.
It's not just about the GIL. Currently, the gc module stores its state into _PyRuntime. It's wrong to share a single gc state between two interpreters: each interpreter should have its own "namespace" completely isolated from the other namespaces. For example, _PyRuntime.gc.garbage is a Python list: each interpreter should have its own list.
My PR 12934 is only a first step to prepare ceval.c for that.
Said differently, if I understood correctly, each interpreter must have its own _PyRuntime instance.
Maybe tomorrow, we will keep a single _PyRuntime instance, *but* my work is needed to identify the relationship between the current implementation of Python and _PyRuntime.
14 remaining items
I started a thread on python-dev about this issue:
"Pass the Python thread state to internal C functions"
https://mail.python.org/archives/list/python-dev@python.org/thread/PQBGECVGVYFTVDLBYURLCXA3T7IPEHHO/I continued this work by passing tstate to internal C functions: bpo-38644.
I also added PyInterpreterState.runtime field, so it's now possible to retrieve the runtime from tstate:
runtime = tstate->interp->runtime;
I wrote an article on passing tstate to internal C functions:
https://vstinner.github.io/cpython-pass-tstate.html
I consider that this issue is now done. I close the issue.
Thanks, Victor!
Instead of passing
_PyRuntimeStatearound everywhere, why not just let it disappear in time.Currently
_PyRuntimeStatemanages "global" state, mainly the GIL and some config.
Once the GIL has been migrated to the sub-interpreters, the config part can be factored out and_PyRuntimeStatecan just disappear.Instead of passing
_PyRuntimeStatearound everywhere, why not just let it disappear in time.Agreed. It's valuable to pass the thread state, but the runtime state should only be needed to create a new thread state (and arguably not even then).
Instead of passing
_PyRuntimeStatearound everywhere, why not just let it disappear in time.Passing runtime (_PyRuntimeState) is a temporary move until more and more fields are moved from _PyRuntimeState into PyInterpreterState. I just created bpo-39984 "Move some ceval fields from _PyRuntime.ceval to PyInterpreterState.ceval" yesterday ;-)
Once we will manage to move the GIL into PyInterpreterState, we would still have to pass PyInterpreterState or PyThreadState to function which require to access the GIL. Passing explicitly runtime is a first step to prepare to migration to PyInterpreterState or PyThreadState. My intent is to show that many functions rely on "global variables": pass these variables instead.
If you are thinking about getting the current Python thread state using a thread local storage, that's a different topic and I'm not aware of an open issue to track this idea.
Currently
_PyRuntimeStatemanages "global" state, mainly the GIL and some config. Once the GIL has been migrated to the sub-interpreters, the config part can be factored out and_PyRuntimeStatecan just disappear.I don't think that we will be able to fully remove _PyRuntimeState. It seems like the PEP-554 "Multiple Interpreters in the Stdlib" requires a registry of interpreters and it currently lives in _PyRuntimeState.
Even if
_PyRuntimeends up as just a list of interpreters and doesn't disappear completely, it won't be used anything like as much as it is now.Many of the functions that it getting passed to will no longer need it, so why bother passing it now?
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: