Repository navigation
test_multiprocessing_fork hangs test_subprocess #75644
Description
| BPO | 31463 |
|---|---|
| Nosy | @pitrou, @vstinner, @benjaminp, @hroncok, @pablogsal |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/pablogsal'
closed_at = <Date 2018-06-27.20:39:46.804>
created_at = <Date 2017-09-14.06:36:00.682>
labels = ['3.7', 'tests']
title = 'test_multiprocessing_fork hangs test_subprocess'
updated_at = <Date 2018-06-27.20:39:46.802>
user = 'https://github.com/benjaminp'bugs.python.org fields:
activity = <Date 2018-06-27.20:39:46.802>
actor = 'vstinner'
assignee = 'pablogsal'
closed = True
closed_date = <Date 2018-06-27.20:39:46.804>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-09-14.06:36:00.682>
creator = 'benjamin.peterson'
dependencies = []
files = []
hgrepos = []
issue_num = 31463
keywords = []
message_count = 15.0
messages = ['302143', '315486', '315487', '315580', '315583', '315584', '315591', '317382', '317399', '320064', '320065', '320066', '320067', '320068', '320617']
nosy_count = 5.0
nosy_names = ['pitrou', 'vstinner', 'benjamin.peterson', 'hroncok', 'pablogsal']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue31463'
versions = ['Python 3.7']
Activity
benjaminp commentedon Sep 14, 2017
This hangs reliably for me on master:
It looks like test_multiprocessing_fork doesn't clean up some of its subprocesses and then test_subprocess hangs on waitpid(0) forever.
hroncok commentedon Apr 19, 2018
This started to bother us in Fedora for various Python versions, so chances are something changed on the system level.
However for us it seams test_multiprocessing_fork hangs by itself, so this might or might not be relevant to
# python3.7 -m test.regrtest test_multiprocessing_fork test_subprocess Run tests sequentially 0:00:00 load avg: 1.16 [1/2] test_multiprocessing_fork /usr/lib64/python3.7/multiprocessing/semaphore_tracker.py:55: UserWarning: semaphore_tracker: process died unexpectedly, relaunching. Some semaphores might leak. warnings.warn('semaphore_tracker: process died unexpectedly, ' Exception in thread Thread-26: Traceback (most recent call last): File "/usr/lib64/python3.7/threading.py", line 917, in _bootstrap_inner self.run() File "/usr/lib64/python3.7/threading.py", line 865, in run self._target(*self._args, **self._kwargs) File "/usr/lib64/python3.7/multiprocessing/resource_sharer.py", line 139, in _serve signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG)) File "/usr/lib64/python3.7/signal.py", line 60, in pthread_sigmask sigs_set = _signal.pthread_sigmask(how, mask) ValueError: signal number 32 out of range(hangs)
Test suite interrupted by signal SIGINT.
2 tests omitted:
test_multiprocessing_fork test_subprocess
Total duration: 1 min 35 sec
Tests result: INTERRUPTED
# python3.7 --version
Python 3.7.0b3
# python3.6 -m test.regrtest test_multiprocessing_fork test_subprocess Run tests sequentially 0:00:00 load avg: 1.84 [1/2] test_multiprocessing_fork /usr/lib64/python3.6/multiprocessing/semaphore_tracker.py:55: UserWarning: semaphore_tracker: process died unexpectedly, relaunching. Some semaphores might leak. warnings.warn('semaphore_tracker: process died unexpectedly, ' Exception in thread Thread-20: Traceback (most recent call last): File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/lib64/python3.6/multiprocessing/resource_sharer.py", line 139, in _serve signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG)) File "/usr/lib64/python3.6/signal.py", line 60, in pthread_sigmask sigs_set = _signal.pthread_sigmask(how, mask) ValueError: signal number 32 out of range(hangs)
Test suite interrupted by signal SIGINT.
2 tests omitted:
test_multiprocessing_fork test_subprocess
Total duration: 51 sec
Tests result: INTERRUPTED
# python3.6 --version
Python 3.6.5
(hangs)
Test suite interrupted by signal SIGINT.
2 tests omitted:
test_multiprocessing_fork test_subprocess
Tests result: INTERRUPTED
# python3.5 --version
Python 3.5.4
(hangs)
Test suite interrupted by signal SIGINT.
2 tests omitted:
test_multiprocessing_fork test_subprocess
# python3.4 --version
Python 3.4.8
hroncok commentedon Apr 19, 2018
Sorry, I've pressed the button before finishing the thought.
...so this might or might not be relevant to what you observe with master and originally reported.
pitrou commentedon Apr 21, 2018
@benjamin
I can't reproduce here. It's true that test_multiprocessing_fork leaves two processes being itself, but that doesn't seem to disturb test_subprocess.
@miro
Could you report the signal number issue separately? Thank you!
benjaminp commentedon Apr 21, 2018
It doesn't hang for me anymore, but does print this warning:
0:00:00 load avg: 1.59 [1/2] test_multiprocessing_fork
Lib/multiprocessing/semaphore_tracker.py:55: UserWarning: semaphore_tracker: process died unexpectedly, relaunching. Some semaphores might leak.
warnings.warn('semaphore_tracker: process died unexpectedly, '
pitrou commentedon Apr 21, 2018
There's a test that deliberately kills the semaphore tracker, so that warning should be expected (I don't see it myself, but that might be because of differing warning settings).
hroncok commentedon Apr 22, 2018
https://bugs.python.org/issue33329
Resetting the Versions back.
vstinner commentedon May 23, 2018
Oh strange. This issue was supposed to be fixed. I spent a lot of time to fix such bug, but also make sure that not test leak any child process.
"test_subprocess hangs on waitpid(0) forever" I recall a similar bug that I fixed recently.
I spent time on test.support.reap_children() and test.support.threading_cleanup() to make sure that tests don't leak threads no child processes. It seems like I missed some tests.
In Lib/test/libregrtest/runtest.py, I added:
which is run after each test.
But currently, support.reap_children() calls os.waitpid(-1, os.WNOHANG) to check for child processes which already completed. It doesn't warn if there are child processes which are still running.
During my tests, I modified reap_children() to add a sleep(1).
Maybe on Linux we could use a cgroup to really check for all child processes?
pitrou commentedon May 23, 2018
Let's take another look at the issue: why does test_subprocess need to know about all child processes, rather than those that were launched during test_subprocess?
pablogsal commentedon Jun 20, 2018
I would like to work on this issue
vstinner commentedon Jun 20, 2018
Last time I was bitten by waitpid():
commit 11045c9
Author: Victor Stinner <victor.stinner@gmail.com>
Date: Thu Oct 5 06:32:53 2017 -0700
vstinner commentedon Jun 20, 2018
See also the now hopefully fixed bpo-31509 "test_subprocess hangs randomly on AMD64 Windows10 3.x".
vstinner commentedon Jun 20, 2018
Antoine: "Let's take another look at the issue: why does test_subprocess need to know about all child processes, rather than those that were launched during test_subprocess?"
It seems like this bug has been fixed by bpo-31178, commit 11045c9, no?
vstinner commentedon Jun 20, 2018
Another old (now fixed) issue of test_subprocess: bpo-31173 "test_subprocess: test_child_terminated_in_stopped_state() leaks a zombie process".
vstinner commentedon Jun 27, 2018
FYI right now, this command doesn't hang anymore.
This bug has been fixed by bpo-33332.
This issue should be fixed by my commit 11045c9.
It seems like all issues described here have been fixed and so I close the issue. If an issue is not properly fixed, maybe it's better to open a new issue since this one has long history and described multiple unrelated bugs.