Skip to content

gh-89727: Fix os.walk to handle late editing of dirs #100703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jonburdo wants to merge 14 commits into from
Closed

gh-89727: Fix os.walk to handle late editing of dirs #100703

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
2767f70
adopt old os.walk behavior to handle changes to dirs list after itera…
jonburdo Jan 3, 2023
04a16a2
move stack logic to bottom of loop in os.walk
jonburdo Jan 3, 2023
6b0465f
move scandir logic to bottom of loop
jonburdo Jan 3, 2023
2ff800f
use for-loop to process dirs in bottom-up clause
jonburdo Jan 3, 2023
072bdfb
Merge branch 'main' into os-walk-dirs-editing
jonburdo Jan 3, 2023
41d7463
return if scandir_it fails on first value
jonburdo Jan 3, 2023
d81ff45
add test for late modification of dirs in os.walk
jonburdo Jan 3, 2023
431d47a
Merge branch 'main' into os-walk-dirs-editing
jonburdo Jan 3, 2023
2829a7d
use while-else instead of while True, make variable names clearer
jonburdo Jan 3, 2023
b59358b
change os.walk docstring to say "traverse" instead of "recurse"
jonburdo Jan 3, 2023
839f31e
fix scandir comment
jonburdo Jan 4, 2023
1011e19
Merge branch 'main' into os-walk-dirs-editing
jonburdo Jan 4, 2023
80b7e82
split topdown and bottomup logic into separate functions for os.walk
jonburdo Jan 4, 2023
70cb0c2
Merge branch 'main' into os-walk-dirs-editing
erlend-aasland Jan 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
use while-else instead of while True, make variable names clearer
  • Loading branch information
@jonburdo
jonburdo committed Jan 3, 2023
commit 2829a7dc11c2ba2e370e9a1d03464966d5019a9c
39 changes: 20 additions & 19 deletions Lib/os.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -355,7 +355,7 @@ def walk(top, topdown=True, onerror=None, followlinks=False):
dirs = []
nondirs = []
walk_dirs = []
cont = False
use_entry = True

# We may not have read permission for top, in which case we can't
# get a list of the files the directory contains.
Expand All @@ -372,7 +372,7 @@ def walk(top, topdown=True, onerror=None, followlinks=False):
except OSError as error:
if onerror is not None:
onerror(error)
cont = True
use_entry = False
break
Comment thread
carljm marked this conversation as resolved.

try:
Expand Down Expand Up @@ -405,19 +405,17 @@ def walk(top, topdown=True, onerror=None, followlinks=False):
if walk_into:
walk_dirs.append(entry.path)
if topdown:

@carljm carljm Jan 4, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Such a high percentage of code in this function is now under either if topdown or the reverse that I'm kind of curious what it would look like to just have separate (internal) functions for topdown vs bottom-up. But this would still increase code duplication significantly, and move the duplicated code/structure further apart, so I suspect it's still better this way.

@jonburdo jonburdo Jan 4, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was actually thinking about this. I just gave it a shot to see. This seems to me like one of those cases where there are enough little differences in logic that it's much cleaner to separate the two. If we didn't support dir modification for topdown or didn't care about performance it might be simpler, but when you have a lot of the same logic interspersed by a lot of little differences it gets messy. In these kinds of situations I also tend to prefer some duplication between functions over one big function with conditions inside of loops.

I also find it easier to look at the two sets of logic separately, but either way works.

@carljm carljm Jan 4, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think the split version seems fine. Will wait a bit and see if some core devs have opinions.

if not cont:
if use_entry:
# Yield before sub-directory traversal if going top down
yield top, dirs, nondirs
# Traverse into sub-directories
# Append dir names to walk along with top, their parent dir
stack.append([top, iter(dirs)])

while True:
try:
value, dirs = stack[-1]
except IndexError:
return
# Set top and scandir_it for the next iteration
while stack:
root, dirs = stack[-1]
for dirname in dirs:
top = join(value, dirname)
top = join(root, dirname)
# bpo-23605: os.path.islink() is used instead of caching
# entry.is_symlink() result during the loop on os.scandir() because
# the caller can replace the directory entry during the previous
Expand All @@ -434,16 +432,18 @@ def walk(top, topdown=True, onerror=None, followlinks=False):
stack.pop()
continue
break
else:
return
else:
if not cont:
# Traverse into sub-directories
if use_entry:
# Traverse into sub-directories by appending a value
# (entry, dirs) where dirs will be walked and then
# when dirs is exhausted entry will be yielded
stack.append(((top, dirs, nondirs), iter(walk_dirs)))

while True:
try:
value, dirs = stack[-1]
except IndexError:
return
# Set top and scandir_it for the next iteration
while stack:
entry, dirs = stack[-1]
for top in dirs:
try:
scandir_it = scandir(top)
Expand All @@ -454,10 +454,11 @@ def walk(top, topdown=True, onerror=None, followlinks=False):
break
else:
stack.pop()
# Yield after sub-directory traversal if going bottom up
yield value
yield entry
continue
break
else:
return

__all__.append("walk")

Expand Down