Repository navigation
test_asyncio environment changed during test execution due to empty threading._dangling without MainThread before some tests #91676
Description
Activity
- changed the title
[-]test environment chained due to empty `threading._dangling` without MainThread before some tests[/-][+]test_asyncio environment changed during test execution due to empty `threading._dangling` without MainThread before some tests[/+]on Apr 18, 2022 I've also seen this one:
../b/python -m test.bisect_cmd test_asyncio.test_threads test_asyncio.test_unix_events --fail-env-changed -j 12 -N 1000 2>&1 | less ... 0:00:00 load avg: 0.64 [1/2/1] test_asyncio.test_threads failed (env changed) Warning -- threading._dangling was modified by test_asyncio.test_threads Warning -- Before: {<weakref at 0x7f67f8967970; to '_MainThread' at 0x7f67f92eb370>} Warning -- After: {<weakref at 0x7f67f91a1210; to 'Thread' at 0x7f67f8979480>, <weakref at 0x7f67f8967f60; to '_MainThread' at 0x7f67f92eb370>}notice the specific weakref instance to the same address
_MainThreadchanged above and here:python -m test --matchfile /tmp/tmpu15fc2a3 --fail-env-changed test_asyncio.test_threads test_asyncio.test_unix_events -j 2 0:00:00 load avg: 1.42 Run tests in parallel using 2 child processes 0:00:00 load avg: 1.42 [1/2/1] test_asyncio.test_threads failed (env changed) Warning -- threading._dangling was modified by test_asyncio.test_threads Before: {<weakref at 0x7f3929113a10; to '_MainThread' at 0x7f3929a93490>} After: {<weakref at 0x7f3929979800; to '_MainThread' at 0x7f3929a93490>} 0:00:00 load avg: 1.42 [2/2/1] test_asyncio.test_unix_events ran no tests == Tests result: ENV CHANGED ==The code in
Lib/test/libregrtest/save_env.pyhas race conditions that could lead to this https://github.com/python/cpython/blob/main/Lib/test/libregrtest/save_env.py#L216 are invoked by enter and exit on thesaved_test_environmentcontext manager. seeresource_infofor how.the irony of code that is supposed to help buildbot instability also being one probable cause of buildbot instability...
The excess threads running on occasion after a test ends are asyncio concurrent.futures threadpool _worker threads named asyncio_0 or asyncio_1...
('<Thread(asyncio_0, started 140097331353152)>', (<function _worker at 0x7f6af3c3f5f0>, '_worker', 'concurrent.futures.thread', None))The change that exacerbated this race condition notably no longer does a
loop.run_until_complete(loop.shutdown_default_executor())which is what I believe would've seen the worker threads cleaned up? [edit: confirmed, PR coming]- linked a pull request that will close this issue[3.10] gh-91676 gh-91260 unittest.IsolatedAsyncioTestCase no longer leaks its executor #91680
on Apr 19, 2022 - linked a pull request that will close this issue[3.9] [3.10] gh-91676 gh-91260 unittest.IsolatedAsyncioTestCase no longer leaks its executor (GH-91680) #91692
on Apr 19, 2022 Thanks for the fix ;-)
We're seeing frequent flakiness on CI and buildbots (heavily loaded systems, triggering more race conditions?) with test_asyncio of late. #91260 has some discussion as that seems to have surfaced it more frequently, at least in the stable 3.10 branch.
from a 3.10 branch with #91674 patched in.
from https://github.com/python/cpython/runs/6069450575?check_suite_focus=true CI and from many stable buildbots such as https://buildbot.python.org/all/#/builders/608/builds/700/steps/6/logs/stdio
While "interesting" for
_MainThreadto wind up inthreading._dangling... code wise it should always be in there. Yet somehow it is not before execution?!?after a fork() from the non-main thread or from C, it won't be. danger - something is using fork with threads running, a guaranteed source of problems. Fixing it to always exist in _dangling beforehand would not be a bad thing, but only if we can figure out how it happens to not be in the first place.