Repository navigation
test_multiprocessing_forkserver.py, test_multiprocessing_fork.py and test_multiprocessing_spawn.py gives tracebacks #96026
Description
Activity
I seen this output many times, but cannot reproduce locally (on Ubuntu).
It is bad because it looks like an error even if test reports OK. It is mixed with real errors report and make it more difficult to analyze. It can also be a symptom of a problem with resource management.
shared_memory.py... try: if create and size: os.ftruncate(self._fd, size) stats = os.fstat(self._fd) size = stats.st_size self._mmap = mmap.mmap(self._fd, size) except OSError: self.unlink() raise resource_tracker.register(self._name, "shared_memory")So if OSError exist, it tries to do
resource_tracker.unregisterwhich is contained inself.unlink(), but maybe it can be situation whenresource_tracker.registerwasn't register before?maybe connected with #89372 ?
I've found the test which reproduce this tracebackpython -m test -v test_multiprocessing_forkserver -m test_shared_memory_cleaned_after_process_termination == CPython 3.11.0rc1 (main, Aug 13 2022, 18:23:13) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] == Linux-3.10.0-1160.62.1.el7.x86_64-x86_64-with-glibc2.17 little-endian == cwd: /tmp/test_python_43253æ == CPU count: 40 == encodings: locale=UTF-8, FS=utf-8 0:00:00 load avg: 21.34 Run tests sequentially 0:00:00 load avg: 21.34 [1/1] test_multiprocessing_forkserver test_shared_memory_cleaned_after_process_termination (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory.test_shared_memory_cleaned_after_process_termination) ... Traceback (most recent call last): File "/opt/rh/rh-python311/root/usr/lib64/python3.11/multiprocessing/resource_tracker.py", line 209, in main ok cache[rtype].remove(name) KeyError: '/psm_60e6d44e' ---------------------------------------------------------------------- Ran 1 test in 0.235s OK == Tests result: SUCCESS == 1 test OK. Total duration: 420 ms Tests result: SUCCESSit is reproducable for
test_multiprocessing_fork,test_multiprocessing_forkserver,test_multiprocessing_spawnIn my case reverting patch f604cf1 doesn't produce any key error and other UserWarning.
So it's like specific forWindowstests?
Checked on current MacOS:python3.10 -m test -v test_multiprocessing_spawn -v -m test_shared_memory_cleaned_after_process_termination Raised RLIMIT_NOFILE: 256 -> 1024 == CPython 3.10.6 (main, Aug 6 2022, 22:42:17) [Clang 13.1.6 (clang-1316.0.21.2.5)] == macOS-12.5-x86_64-i386-64bit little-endian == cwd: /private/var/folders/pr/4ktlw3_s2bxfb3ggg496dc1h0000gp/T/test_python_92918æ == CPU count: 8 == encodings: locale=UTF-8, FS=utf-8 0:00:00 load avg: 1.90 Run tests sequentially 0:00:00 load avg: 1.90 [1/1] test_multiprocessing_spawn test_shared_memory_cleaned_after_process_termination (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ... ok unregister /psm_9b1c91a5 Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/resource_tracker.py", line 211, in main cache[rtype].remove(name) KeyError: '/psm_9b1c91a5' ---------------------------------------------------------------------- Ran 1 test in 0.237s OK == Tests result: SUCCESS == 1 test OK. Total duration: 373 ms Tests result: SUCCESSit tries to run
resource_tracker.unregisteron the resource which wasn't added into resource_tracker before.0:00:00 load avg: 20.19 [1/1] test_multiprocessing_spawn test_shared_memory_cleaned_after_process_termination (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory.test_shared_memory_cleaned_after_process_termination) ... b'REGISTER:/mp-kxz5k9_d:semaphore\n' register /mp-kxz5k9_d b'PROBE:0:noop\n' b'UNREGISTER:/mp-kxz5k9_d:semaphore\n' unregister /mp-kxz5k9_d resource_tracker.unregister name is: psm_674b5480 b'PROBE:0:noop\n' b'UNREGISTER:/psm_674b5480:shared_memory\n' unregister /psm_674b5480 Traceback (most recent call last): File "/opt/rh/rh-python311/root/usr/lib64/python3.11/multiprocessing/resource_tracker.py", line 212, in main ok cache[rtype].remove(name) KeyError: '/psm_674b5480' ---------------------------------------------------------------------- Ran 1 test in 0.220sI'm not entirely confident yet, but it appears that after 9a458be got merged, the workaround from f604cf1 is no longer needed (and causes the reported issue).
@tiran is it possible to fix it by reverting workaround as mentioned in comment by @sergeykolosov ?
@sharewax Please refrain from pinging core developers on issues to draw their attention. It's distracting. Triagers may ping core devs or assign tickets that are listed as experts in https://devguide.python.org/core-developers/experts/index.html.
sorry for that :(
Also encountered this problem for Python 3.9.21 on macOS arm64.
- added a commit that references this issue
on Apr 30, 2025
Bug report
During build RPM for python3.11rc1 I've found tracebacks for tests from test suite
test_multiprocessing_forkserver.py,test_multiprocessing_fork.py, andtest_multiprocessing_spawn.py:I suppose, that tracebacks are not ok during test. Or close the bug, if it normal.
Some
KeyErrorare present in #95396.Your environment
Python 3.11.0rc1 (main, Aug 13 2022, 18:23:13) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux
Centos7 x86_64, python 3.11rc1