Repository navigation
Everyone uses GIL wrong! = DEADLOCK #81367
Description
| BPO | 37186 |
|---|---|
| Nosy | @vstinner, @ericsnowcurrently, @Roffild |
| Superseder |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-06-07.17:29:01.707>
created_at = <Date 2019-06-07.00:03:04.799>
labels = ['interpreter-core', '3.7', 'type-crash']
title = 'Everyone uses GIL wrong! = DEADLOCK'
updated_at = <Date 2019-06-07.17:29:01.703>
user = 'https://github.com/Roffild'bugs.python.org fields:
activity = <Date 2019-06-07.17:29:01.703>
actor = 'eric.snow'
assignee = 'none'
closed = True
closed_date = <Date 2019-06-07.17:29:01.707>
closer = 'eric.snow'
components = ['Interpreter Core']
creation = <Date 2019-06-07.00:03:04.799>
creator = 'Roffild'
dependencies = []
files = []
hgrepos = []
issue_num = 37186
keywords = []
message_count = 2.0
messages = ['344891', '344974']
nosy_count = 3.0
nosy_names = ['vstinner', 'eric.snow', 'Roffild']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '10915'
type = 'crash'
url = 'https://bugs.python.org/issue37186'
versions = ['Python 3.7']
Activity
Roffild commentedon Jun 7, 2019
Everyone uses GIL wrong! = DEADLOCK
I used sub-interpreters in embedded Python:
https://github.com/Roffild/RoffildLibrary/blob/35ef39fafc164d260396b39b28ff897d44cf0adb/Libraries/Roffild/PythonDLL/private.h#L44
https://github.com/Roffild/RoffildLibrary/blob/35ef39fafc164d260396b39b28ff897d44cf0adb/Libraries/Roffild/PythonDLL/mql_class.c#L142
A deadlock happens in the line:
cpython/Python/pystate.c
Line 1313 in 7114c65
Of course in the help there is the note:
Note that the PyGILState_() functions assume there is only one global interpreter (created automatically by Py_Initialize()). Python supports the creation of additional interpreters (using Py_NewInterpreter()), but mixing multiple interpreters and the PyGILState_() API is unsupported.
But functions PyGILState_() are used in third-party libraries. Most often, these functions are used without checking that GIL is already locked. Often, these functions are added to the code for reinsurance only and this can affect performance.
Numpy:
https://github.com/numpy/numpy/blob/2d4975e75c210202293b894bf98faf12f4697a31/numpy/core/include/numpy/ndarraytypes.h#L987
https://github.com/numpy/numpy/search?q=NPY_ALLOW_C_API&unscoped_q=NPY_ALLOW_C_API
Pytorch:
https://github.com/pytorch/pytorch/blob/0a3fb45d3d2cfacbd0469bbdba0e6cb1a2cd1bbe/torch/csrc/utils/auto_gil.h#L9
https://github.com/pytorch/pytorch/search?q=AutoGIL&unscoped_q=AutoGIL
Pybind11 developers have already fixed this problem:
https://github.com/pybind/pybind11/blob/97784dad3e518ccb415d5db57ff9b933495d9024/include/pybind11/pybind11.h#L1846
It is necessary to change the code of PyGILState_() functions to support sub-interpreters.
Or add to https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock warning:
Some Python libraries cannot be used in a sub-interpreter due to the likelihood of deadlock.
For me, this is a critical vulnerability!
There is another problem:
Calling PyEval_AcquireThread() again results in a deadlock. This can be controlled in your code, but not in a third-party library.
ericsnowcurrently commentedon Jun 7, 2019
The situation with subinterpreters and the gilstate API is a known problem (see issues bpo-10915 and bpo-15751). We plan on fixing that in the near future for 3.9 (and probably for 3.8). Feel free to chime in on those earlier issues. Note that the subinterpreter experience is going to be much improved with the 3.9 release.
You also mentioned an issue with PyEval_AcquireThread(). If you don't think this is directly related to the gilstate API (I suspect it is) then please open a new issue for that.