Skip to content

bpo-29694: race condition in pathlib mkdir with flags parents=True #1089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Mariatta merged 4 commits into from
Apr 13, 2017
Merged

bpo-29694: race condition in pathlib mkdir with flags parents=True #1089

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d1a214a
Issue #29694: race condition in pathlib mkdir with flags parents=True
arigo Apr 12, 2017
f77b3db
Fixes from feedback (thanks Serhiy)
arigo Apr 12, 2017
29af33b
typo
arigo Apr 13, 2017
a3397be
Add `Patch By Armin Rigo` to Misc/NEWS
Mariatta Apr 13, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Lib/pathlib.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1217,8 +1217,8 @@ def mkdir(self, mode=0o777, parents=False, exist_ok=False):
except FileNotFoundError:
if not parents or self.parent == self:
raise
self.parent.mkdir(parents=True)
self._accessor.mkdir(self, mode)
self.parent.mkdir(parents=True, exist_ok=True)
self.mkdir(mode, parents=False, exist_ok=exist_ok)
except OSError:
# Cannot rely on checking for EEXIST, since the operating system
# could give priority to other errors like EACCES or EROFS
Expand Down
31 changes: 31 additions & 0 deletions Lib/test/test_pathlib.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import stat
import tempfile
import unittest
from unittest import mock

from test import support
android_not_root = support.android_not_root
Expand Down Expand Up @@ -1801,6 +1802,36 @@ def test_mkdir_no_parents_file(self):
p.mkdir(exist_ok=True)
self.assertEqual(cm.exception.errno, errno.EEXIST)

def test_mkdir_concurrent_parent_creation(self):
for pattern_num in range(32):
p = self.cls(BASE, 'dirCPC%d' % pattern_num)
self.assertFalse(p.exists())

def my_mkdir(path, mode=0o777):
path = str(path)
# Emulate another process that would create the directory
# just before we try to create it ourselves. We do it
# in all possible pattern combinations, assuming that this
# function is called at most 5 times (dirCPC/dir1/dir2,
# dirCPC/dir1, dirCPC, dirCPC/dir1, cirCPC/dir1/dir2).

@berkerpeksag berkerpeksag Apr 12, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cirCPC looks like a typo here.

if pattern.pop():
os.mkdir(path, mode) # from another process
concurrently_created.add(path)
os.mkdir(path, mode) # our real call
org_mkdir = pathlib._normal_accessor.mkdir

@serhiy-storchaka serhiy-storchaka Apr 12, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems this is not used.


pattern = [bool(pattern_num & (1 << n)) for n in range(5)]
concurrently_created = set()
p12 = p / 'dir1' / 'dir2'
try:
with mock.patch("pathlib._normal_accessor.mkdir", my_mkdir):
p12.mkdir(parents=True, exist_ok=False)
got_exception = False
except FileExistsError:
got_exception = True
self.assertEqual(str(p12) in concurrently_created, got_exception)

@serhiy-storchaka serhiy-storchaka Apr 12, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assertIn()/assertNotIn()? This will give more information in the case of failure.

try:
    ...
except FileExistsError:
    self.assertIn(str(p12), concurrently_created)
else:
    self.assertNotIn(str(p12), concurrently_created)

self.assertTrue(p.exists())

@support.skip_unless_symlink
def test_symlink_to(self):
P = self.cls(BASE)
Expand Down
3 changes: 3 additions & 0 deletions Misc/NEWS
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,9 @@ Extension Modules

Library
-------

- bpo-29694: race condition in pathlib mkdir with flags parents=True

@serhiy-storchaka serhiy-storchaka Apr 12, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add "Fixed" at the start, add a period at the end.


- bpo-29692: Fixed arbitrary unchaining of RuntimeError exceptions in
contextlib.contextmanager.
Patch by Siddharth Velankar.
Expand Down