Skip to content

bpo-33613, test_semaphore_tracker_sigint: fix race condition #7850

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pitrou merged 21 commits into from
Sep 4, 2018
Merged

bpo-33613, test_semaphore_tracker_sigint: fix race condition #7850

Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5f1df08
bpo-33613, test_semaphore_tracker_sigint: fix race condition
pablogsal Jun 20, 2018
4f9babb
Implement PONG command in the semaphore tracker
pablogsal Jun 24, 2018
26d81a3
Ensure the tracker is killed to allow multiple testing runs in the sa…
pablogsal Jun 24, 2018
a7d5c59
fixup! Ensure the tracker is killed to allow multiple testing runs in…
pablogsal Jul 1, 2018
a735d49
fixup! fixup! Ensure the tracker is killed to allow multiple testing …
pablogsal Jul 7, 2018
9460da4
fixup! fixup! fixup! Ensure the tracker is killed to allow multiple t…
pablogsal Jul 24, 2018
fb136d0
Use a buffered writer to answer to the PING command
pablogsal Jul 24, 2018
f3d0f0b
Use the tracker own stderr for answering the PING command
pablogsal Jul 24, 2018
f5e5cf9
Use pthread_sigmask to avoid the race condition
pablogsal Jul 27, 2018
4393021
Add comment explaining the race condition
pablogsal Jul 27, 2018
d501f2f
Restore timeout and use os.waitpid
pablogsal Jul 31, 2018
9e0243e
fixup! Restore timeout and use os.waitpid
pablogsal Jul 31, 2018
26e077b
Add News entry
pablogsal Jul 31, 2018
a50912b
fixup! fixup! Restore timeout and use os.waitpid
pablogsal Jul 31, 2018
bcbb942
fixup! Add News entry
pablogsal Jul 31, 2018
0b46f07
fixup! fixup! fixup! Restore timeout and use os.waitpid
pablogsal Aug 1, 2018
66ef7dc
Unregister sigmask if process fails to start
pablogsal Aug 3, 2018
f7d2d4d
Include the signal registering in the try block
pablogsal Sep 3, 2018
4a09fde
Simplify comment when handling ChildProcessError
pablogsal Sep 3, 2018
c9f96c1
Eliminate hardcoded reference to the bug tracker url
pablogsal Sep 3, 2018
c0a088c
Remove extraneous NEWS file
pitrou Sep 4, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Ensure the tracker is killed to allow multiple testing runs in the sa… 
…me process
  • Loading branch information
@pablogsal
pablogsal committed Jun 24, 2018
commit 26d81a39959bb650364e755e30ec653f5b5b81ad
9 changes: 7 additions & 2 deletions Lib/test/_test_multiprocessing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4473,19 +4473,24 @@ def check_semaphore_tracker_death(self, signum, should_die):
# bpo-31310: if the semaphore tracker process has died, it should
# be restarted implicitly.
from multiprocessing.semaphore_tracker import _semaphore_tracker
pid = _semaphore_tracker._pid
if pid:

@vstinner vstinner Jul 31, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like the multiprocessing code uses "pid is not None". Technically, I'm not sure that it's possible to have a pid of 0, since pid 0 has a special meaning in many C functions.

@pablogsal pablogsal Jul 31, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use pid is not None here. I think is clearer what the value you are expecting not to be.

@vstinner vstinner Jul 31, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm now confused, you modified "if not pid" in the other file, whereas it seems like os.waitpid() doesn't return None.

@pablogsal pablogsal Aug 1, 2018 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I did not understand correctly your suggestion. For some reason, I inferred that with WNOHANG, waitpid could return None (as waitid can do if you provide WNOHANG). I think I understand now. Mea culpa :)

@pablogsal pablogsal Aug 1, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 822516a1403d4eaec9ac743861336f0902d8186e

os.kill(pid, signal.SIGKILL)
time.sleep(0.5) # give it time to die

@vstinner vstinner Jul 27, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to wait until the process completes instead of using a sleep?

@pablogsal pablogsal Jul 27, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sadly no AFAIK, because the tracker relies on calling waitpid to check the child. If you add a os.waitpid here you will get:


ERROR: test_semaphore_tracker_sigint (test.test_multiprocessing_fork.TestSemaphoreTracker)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/pgalindo3/cpython/Lib/test/_test_multiprocessing.py", line 4509, in test_semaphore_tracker_sigint
    self.check_semaphore_tracker_death(signal.SIGINT, False)
  File "/Users/pgalindo3/cpython/Lib/test/_test_multiprocessing.py", line 4481, in check_semaphore_tracker_death
    _semaphore_tracker.ensure_running()
  File "/Users/pgalindo3/cpython/Lib/multiprocessing/semaphore_tracker.py", line 48, in ensure_running
    pid, status = os.waitpid(self._pid, os.WNOHANG)
ChildProcessError: [Errno 10] No child processes
Ran 1 test in 0.068s

@pablogsal pablogsal Jul 27, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can make the child capture ChildProcessError on that waitpid and treat that as if the child has died, making it possible to call waitpid in the test.

@vstinner vstinner Jul 31, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not using os.waitpid() in the test, and modify ensure_running() to handle ChildProcessError? The purpose of the waitpid() call in ensure_running() is to check if the process died. We don't care of the exit status.

Adding time.sleep(0.5) means adding a new race condition, while fixing another, no?

old_stderr = sys.stderr
r, w = os.pipe()
try:

@pablogsal pablogsal Jun 24, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice that we cannot use test.support.captured_stderr() as it does not support fileno().

sys.stderr = open(w, "bw")

@taleinat taleinat Jun 30, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line should be before the try:.

@pablogsal pablogsal Jul 1, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d075473

_semaphore_tracker.ensure_running()
with warnings.catch_warnings(record=True) as all_warn:
_semaphore_tracker.ensure_running()
pid = _semaphore_tracker._pid
# Wait until we receive the PONG from the child, indicating that
# the signal handlers have been registered. See bpo-33613 for more
# information.
_semaphore_tracker._send("PING", "")
with open(r, "rb") as pipe:
data = pipe.readline()

@taleinat taleinat Jun 30, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we okay with this potentially hanging indefinitely if something goes wrong?

@pablogsal pablogsal Jul 1, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit d075473 implements a way to fail if reading takes too long.

if b"PON" not in data:
if b"PONG" not in data:
raise ValueError("Invalid data in stderr!")
finally:
sys.stderr.close()
Expand Down