Repository navigation
pathlib.Path.iterdir doesn't raise an exception until you start iterating #78722
Description
Activity
PaulPinterits commented
on Aug 29, 2018 PaulPinteritsmannequinMannequinAuthorThe fact that
Path.iterdir()only throws exceptions once you start iterating over it makes it very difficult to write correct code.Let's look at an example: We'll iterate over all children of a directory and print their file size.
If we try to do it like this, the
try...excepthas no effect whatsoever:try: children = path_that_doesnt_exist.iterdir() except FileNotFoundError: print("directory doesn't exist") for path in children: print(path.stat().st_size)
If we explicitly check whether the path exists (and is a directory), we end up with a race condition:
if path_that_doesnt_exist.is_dir(): print("directory doesn't exist") else: for path in children: print(path.stat().st_size)
We can wrap the whole loop in a
try...except, but then we might end up catching exceptions we didn't intend to catch. (For example, the exception that's thrown when we try to get the size of a broken symlink.)try: for path in path_that_doesnt_exist.iterdir(): print(path.stat().st_size) # this can also throw FileNotFoundError except FileNotFoundError: print("directory doesn't exist")
We can manually call
nexton the iterator inside of atry..exceptblock, but that's awfully verbose and requires knowledge about iterators and thenextfunction:children = iter(path_that_doesnt_exist.iterdir()) while True: try: path = next(children) except FileNotFoundError: print("directory doesn't exist") break print(path.stat().st_size)
Or we can turn the iterator into a list inside of a
try...except, which seems to be the best option, but completely defeats the point of having an iterator:try: children = list(path_that_doesnt_exist.iterdir()) except FileNotFoundError: print("directory doesn't exist") else: for path in children: print(path.stat().st_size)
As you can see, writing correct (and good) code with
iterdiris more difficult than it has any right to be. Please change this behavior so that exceptions are thrown immediately wheniterdiris called.PaulPinterits commented
on Aug 29, 2018 PaulPinteritsmannequinMannequinAuthorAs an afterthought, I'd like to suggest an alternative solution: If changing the
iterdirbehavior is not possible or not desirable for some reason, please add aPath.listdirmethod that returns a list instead of an iterator. Lazy file system operations can often introduce unnecessary race conditions in the code, so I believe it would be very useful if every lazy method had an eager equivalent.prudvinit commented
on Aug 29, 2018 prudvinitmannequinMannequinMade changes, pathlib.Path('.').iterdir() now throws a FileNotFoundError if the path is not valid.
Work-around: call
list(path.iterdir()):try: children = list(path_that_doesnt_exist.iterdir()) except FileNotFoundError: print("directory doesn't exist") for path in children: print(path.stat().st_size)
I don't think we can change the
iterdir()behaviour at this stage, nor does it justify the introduction of a similar method likelistdir()to my mind.Reacted by C.A.M. GerlachYeah, the fact that it returns a generator/lazy iterator seems pretty fundamental to
iterdir's behavior, and matches the other current and pastiter*functions/methods in Pythons, as well aspathlib.globand similar methods (as well as builtins likerangeand others) which would make it suddenly behaving eagerly quite surprising to many/most users, as well as sacrificing performance/efficiency.list(<lazy iterator>)to make something iterable is a common idiom in Python to handle this situation (and others), and adding a separate method to do the same thing eagerly seems thoroughly redundant.I'm going to close this issue per CAM's rationale above. Use
list(path.iterdir())if you need to handle exceptions before you iterate. Thanks all.Reacted by C.A.M. GerlachI don't believe
list(iterdir())is a suitable workaround, as it requires the entire directory to be materialized in memory, increasing storage with little benefit. I'd argue that itertools (or similar) should provide a mechanism to filter or stop on exceptions, so consuming the iterable can continue to be lazy.more_itertools.iter_except almost handles this case, but (a) requires the callable to be passed and (b) can't handle
StopIterationas a stop condition.Reacted by Barney GaleUpon reflection I agree. I've put a PR up. Do you have a preference on whether it should be backported?
- changed the title
[-]pathlib.Path.iterdir doesn't throw an exception until you start iterating[/-][+]pathlib.Path.iterdir doesn't raise an exception until you start iterating[/+]on Aug 28, 2023 A comment from @AA-Turner:
I agree fixing iterdir at the source would be cleaner
- added a commit that references this issue
on Sep 2, 2023 Thanks for the fix!
Upon reflection I agree. I've put a PR up. Do you have a preference on whether it should be backported?
This feels like a change in behavior / improvement, so probably not suitable for a backport.
Reacted by Barney Gale
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs
pathlib.Path.iterdir()without delay. #107320