Repository navigation
Optimize pathlib.PurePath.__fspath__() #102783
Description
Feature or enhancement
Return an unnormalized path from pathlib.PurePath.__fspath__()
Pitch
Code like open(Path('./README.txt')) or Path('/home//barney').iterdir() shouldn't require us to normalize the path in pathlib (e.g. remove . segments, doubled slashes, etc), as OS APIs are perfectly happy with unnormalized paths.
We can improve the performance of most Path methods, and the effective performance of passing a Path object to any API that accepts os.PathLike, by skipping normalization in __fspath__().
Prerequisites
Pathlib must not normalize paths on construction:
Pathlib's normalization must not change the meaning of paths:
Activity
eryksun commentedon Mar 17, 2023
On Windows, an exception should be made to normalize extended paths in
__fspath__(). If an extended path isn't normalized, the open will fail. One needs to replace slashes with backslashes; collapse repeated slashes; and resolve "." and ".." components.Scripts have to be prepared to handle extended paths coming from
os.path.realpath(),__file__, and attributes of thesysmodule such asexecutable,prefix,argv, andpath. Extended paths can also come from extension modules that use API functions such asGetModuleFileNameW()(if an EXE/DLL was loaded using an extended path),GetFinalPathNameByHandleW(), and any of thePathCch*()functions that implements the flagPATHCCH_ALLOW_LONG_PATHS.For path construction, there's an open issue that proposes to modify
ntpath.join()to fully normalize any drive path or UNC path vianormpath(). For paths without a drive, it's proposed to replace slashes with backslashes and collapse repeated slashes. For example, this change would support getting a useful result from something likentpath.join(filename, "../spam//eggs.py")givenfilenameis an extended path such as "\\?\C:\foo\bar.py".barneygale commentedon Mar 17, 2023
I don't think pathlib's normalization should change the meaning of paths. It sounds like we should adjust the normalization routine to preserve forward slashes, repeated slashes, and "." components, for these kinds of path. ".." components are already be preserved I think.
eryksun commentedon Mar 17, 2023
In almost all cases, extended paths are used solely to access a long path. The need to get a literal path is rare, especially in regard to slashes and "." and ".." component names. Microsoft's filesystems will fail an open that has forward slashes or repeated backslashes. NTFS will fail an open that has "." or ".." component names. FAT filesystems allow creating "." and ".." component names as a file's long name, with some other random short name, but it's very dysfunctional.
People assume that
open(filename / "../spam//eggs.txt")will just work. But iffilenameis an extended path -- one that's injected completely outside of the control of the script -- then this open will fail if the path isn't normalized. IMO, no purity in regards to literal paths is worth this headache.pitrou commentedon Apr 20, 2023
What happens if a path is mounted on e.g. a FUSE filesystem or a Samba share? Does the OS unnormalize before handing the path to the filesystem?
pythonGH-102783: Speed up `pathlib.PurePath.__fspath__()` by returnin…
barneygale commentedon Dec 19, 2023
Resolving as "can't do", because #65238 is resolved as "won't fix"