Add os.path.isrelative() and improve ntpath.isabs()

@mhammond

BPO	1669539
Nosy	@mhammond, @terryjreedy, @tiran, @tjguk, @ezio-melotti, @bitdancer, @zware, @serhiy-storchaka
Dependencies	bpo-15414: os.path.join behavior on Windows (ntpath.join) is not well documented
Files	win32_absolute_paths.patch: Patch v1 win32_absolute_paths_2.patch: Patch v2 - with docs

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2007-02-26.23:07:51.000>
labels = ['type-feature', 'library']
title = 'Add os.path.isrelative() and improve ntpath.isabs()'
updated_at = <Date 2019-04-26.17:48:31.669>
user = 'https://bugs.python.org/jongfoster'

bugs.python.org fields:

activity = <Date 2019-04-26.17:48:31.669>
actor = 'BreamoreBoy'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2007-02-26.23:07:51.000>
creator = 'jongfoster'
dependencies = ['15414']
files = ['7798', '7799']
hgrepos = []
issue_num = 1669539
keywords = ['patch']
message_count = 15.0
messages = ['51978', '51979', '51980', '51981', '51982', '51983', '51984', '58743', '110297', '117582', '138215', '166066', '218560', '222855', '229053']
nosy_count = 11.0
nosy_names = ['mhammond', 'terry.reedy', 'jorend', 'christian.heimes', 'tim.golden', 'ezio.melotti', 'eckhardt', 'r.david.murray', 'zach.ware', 'serhiy.storchaka', 'mdengler']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1669539'
versions = ['Python 3.5']

Linked PRs

GH-44626, GH-105476: Fix ntpath.isabs() handling of part-absolute paths #113829

Hi,

I consider this to be a bug in os.path.isabs():

PythonWin 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32.
>>> import os.path
>>> s = '\\Python25'
>>> os.path.isabs(s)
True
>>> os.path.abspath(s)
'C:\\Python25'
>>> os.chdir('d:')
>>> os.path.abspath(s)
'D:\\Python25'
>>>

If s is really an absolute path as isabs() claims, then why does abspath() return a different path (i.e. not s)? And worse, note that a call to os.chdir() changes the meaning of s! So s is clearly not an absolute path, and isabs() is wrong.

It turns out that on Windows there are really 4 different kinds of paths:

Completely relative, e.g. foo\bar
Completely absolute, e.g. c:\foo\bar or \\server\share
Halfbreeds with no drive, e.g. \foo\bar
Halfbreeds relative to the current working directory on a specific drive, e.g. c:foo\bar

Python 2.5's os.path.isabs() method considers both (2) and (3) to be absolute; I agree with the classification of (2) but strongly disagree about case (3).

Oh, and os.path.join is also broken, e.g. os.path.join('foo', 'a:bar') gives 'foo\\a:bar', which is an invalid path.

Another consequence of this is that os.path.isabs() is not enough to classify paths. Sometimes you really need a relative path, so we really need (at least) a new os.path.isrelative(), which can return "not isabs(s)" on POSIX platforms, and do the right thing on Win32.

The attached patch:

Changes the behaviour of os.path.isabs() and os.path.join() on Win32, to classify pathnames as described above (including adding UNC support)
Adds os.path.isrelative() on all platforms
Changes a couple of Win32 os.path tests where I have deliberately broken backward compatibility
Adds lots of new tests for these 3 functions on Win32

This does change the behaviour of isabs(), and it is possible that existing applications might be depending on the current behaviour. Silently changing the behaviour might break those applications. I'm not sure what the best course of action is - applying this for 2.6, putting a new API in (os.path.isreallyabs()?), or waiting for Python 3000...?

Kind regards,

Jon Foster

P.S. The patch is against Python SVN trunk

Thanks for submitting this. Nice summary of the different cases. I think very few people have given this that much thought. :)

You need a patch for the docs if you want this to even get considered. Then the main obstacle is convincing the team that the nice new logical behavior is worth the breakage. I would be surprised if it got through.

And Python 3000 is not going to be the "subtle breakage" release. I would give you better odds if your patch created new APIs instead, and *documented* exactly what's useful about the model you're suggesting.

There is an interesting thread on python-dev right now about possibly tweaking the behavior of splitext() in 2.6. So maybe there is hope after all.

Thanks for the comments. Updated patch attached, which includes the doc changes.
File Added: win32_absolute_paths_2.patch

I agree the current behaviour is not ideal - but the right way forward isn't clear. Note the existing (strange) behaviour of os.path.join:

>>> os.path.join("c:\\", "\\a")
'c:\\a'
>>> os.path.join("c:\\foo", "\\a")
'\\a'

if isabs('\\a') returns False, there is almost an expectation that the last example should return 'c:\\foo\\a' - but that would almost never give what was expected. 'c:\\a' seems the correct result, but my quick scan of the test code in the patch shows this isn't exercised.

FWIW, about the only common use of isabs() in my code is something like:

log_directory = get_option_from_config_file(...)
if not os.path.isabs(log_directory):
  # make log_directory absolute based on app directory.
  log_directory = os.path.join(app_directory, log_directory)

Best I can tell, the above would not break if isabs('/foo') started returning False, so long as os.path.join('c:\\foo', '\\bar') resulted in 'c:\\bar'.

Off the top-of-my-head, another strategy may be to introduce isrelative() in 2.6, and in 2.7 introduce a warning when isabs() is passed one of the halfbreeds, warning that 2.8 will return a different value, and pointing to the new isrelative(). 2.8 then gets the full, 'correct' implementation as described here.

Hi Mark,

Thanks for reviewing this.

You wrote:

os.path.join("c:\\foo", "\\a")
...
'c:\\a' seems the correct result

Yes indeed, and this patch makes it do that. Hiding in the patch is this changed test case:

-tester("ntpath.join('c:/a', '/b')", '/b')
+tester("ntpath.join('c:/a', '/b')", 'c:/b') # CHANGED: Was '/b')

and these new ones:

+tester('ntpath.join("c:\\a", "\\b")', "c:\\b")
+tester('ntpath.join("c:/a", "/b")', "c:/b")

(Hmm looks like one of those tests is a duplicate... not a big deal I guess).

I guess another way to handle these changes is to introduce new isabsolute(), isrelative(), and joinpath() functions, and deprecate isabs() and join(). (By "deprecate" I mean at least put a warning in the docs that it shouldn't be used; whether or not to make the code raise a DeprecationWarning is a separate decision). But that's kinda ugly from an API design point of view. I'm happy to roll a patch that does that, if we decide that fixing isabs() isn't possible.

It's a pity there isn't a way to do something like "from __future__ import fixed_win_paths". But given how dynamic Python is, I guess that would be hard.

Kind regards,

Jon

The problem should be addressed and fixed before the next alpha release.

Nosy list changed as both have an interest in Windows issues.

I just stumbled across the issue with isabs(). I'd also say that Mark Hammond already provided the typical use case for this, i.e. that you allow relative paths for convenience when storing them or putting them on the commandline, but for actual use you first convert them so that their meaning doesn't change under your feet, e.g. because they depend on volatile things like the current working directory or drive.

If you assume that is the intention or typical use case, then calling isabs() to determine if the path is stable doesn't work. Of course, the wording of the documentation then needs to change, too, as it explicitly says "after chopping off a potential drive letter".

Concerning the behaviour of path.join() and support for "\\server\share" paths, I'm not sure, but I think that these are issues that can be discussed/changed separately.

A bug for tracker purposes is a discrepancy between doc and code. That can be fixed in current versions. A design change is a feature request and can only go in future versions. A deprecation warning for one cycle is desirable when appropriate.

"os.path.isabs(path)
Return True if path is an absolute pathname. On Unix, that means it begins with a slash, on Windows that it begins with a (back)slash after chopping off a potential drive letter."

From what you report, the code does just that, so there is no bug.
Breaking code just because you do not like the current definition is very unlikely to be accepted.

Adding a .isrelative(path) that seems to be equivalent to 'not .ispath' is unlikely to accepted. People who care about the presence of a drive letter can write "path[1]==':'"

"os.path.join(path1[, path2[, ...]])
Join one or more path components intelligently. If any component is an absolute path, all previous components (on Windows, including the previous drive letter, if there was one) are thrown away, and joining continues."

>>> import os
>>> os.path.join('foo', 'a:bar')
'foo\\a:bar'

Producing a non-path does not seem intelligent, so I agree that this a bug. I am not sure what this should produce. 'foo/bar', 'a:foo/bar', and raising an exception would be candidates. All would be better than the current useless return. In any case, a bug fix patch can only fix bugs and should not be intermixed with feature patches.

As to the second sentence: your proposal to change documented and inplemented semantics is, again, a feature request, not a bug fix.

If you want to continue with your feature requests, you can if you wish reclassify this issue, retitle again, and open a new behavior issue for the path bug and upload a bugfix-only patch. Before or along with doing so, I would suggest discussion on the python-ideas list.

pythonGH-44626, pythonGH-105476: Fix `ntpath.isabs()` handling of par…

py313: Fix root paths in tests on Windows

In pypa/setuptools#4669, we learned that this change has broken some assumptions (as intended, I guess), but it leaves the user without a useful "is absolute even if not fully qualified" check.

I'm also encouraged to this view because, as you noted, the hypothetical isabs(path, strict=None) (current behaviour) isn't particularly useful.

That function would be useful in the distutils case, where the caller is interested in making sure the path is relative (i.e. no leading slash). By changing the meaning of isabs, it's now the responsibility of downstream to implement is_drive_absolute where needed and to adapt across Python versions.

Personally, I felt the old isabs behavior was preferable, because it gave a more consistent behavior across platforms. I prefer to find interfaces that bridge the gaps between Windows and non-Windows systems rather than those that reflect the idiosyncrasies. I like to treat my Windows systems much like Unix systems, having only one drive and mounting new disks to other paths on that C drive, such that \foo really can be treated as absolute in most cases.

Shouldn't a backward-incompatible change like this have gone through a deprecation period, warning when a drive-relative absolute path was passed to isabs on Windows (e.g. isabs('/'))? In the current state, developers now have to contend with variance across the matrix of platform and Python version.

Perhaps this change should be reverted and instead a replacement created, something like isreallyabs (or ispureabs), which could be used by the caller when isabs is insufficient?

Probably distutils will switch to use pathlib for this "make relative" function to get compatibility across Python versions. I'm hoping it's no more than the two lines of code that already exist.

but it leaves the user without a useful "is absolute even if not fully qualified" check.

That use case is rare enough that I don't think it deserves a spot in os.path personally. I try to steer people away from part-absolute paths (e.g. \foo or C:foo) because they're too easy to misunderstand.

That function would be useful in the distutils case, where the caller is interested in making sure the path is relative (i.e. no leading slash).

Isn't distutils also interested in whether the path has a drive letter? os.path.splitroot() would work here, but it's 3.12+ only:

def is_relative(path):
    drive, root, _tail = os.path.splitroot(path)
    return not root and not drive

Personally, I felt the old isabs behavior was preferable, because it gave a more consistent behavior across platforms. I prefer to find interfaces that bridge the gaps between Windows and non-Windows systems rather than those that reflect the idiosyncrasies. I like to treat my Windows systems much like Unix systems, having only one drive and mounting new disks to other paths on that C drive, such that \foo really can be treated as absolute in most cases.

I agree with most of this, but I reached a different conclusion. For me, the cross-platform definition of "absolute path" is roughly "unaffected by context", i.e. independent of the current drive or directory. For cross-platform code I try to leave the details to os.path rather than using POSIX-style paths throughout and never changing to another Windows drive.

FWIW, the new behaviour also agrees with pathlib.PurePath.is_absolute().

Shouldn't a backward-incompatible change like this have gone through a deprecation period, warning when a drive-relative absolute path was passed to isabs on Windows (e.g. isabs('/'))? In the current state, developers now have to contend with variance across the matrix of platform and Python version.

Maybe it should have. We skipped backporting to reduce disruption, but perhaps that's not enough.

Should we revert this change, and instead emit a (deprecation?) warning when a drive-less rooted path is given to ntpath.isabs()? Penny for any opinions further to Jason's and my own.

In distutils, I ended up re-implementing the make_relative logic with pathlib.Path.relative_to(pathlib.Path.anchor), so the compatibility concerns are now moot for that use-case. I'm unsure how many other cases are affected by this change. Perhaps it's best to leave it as implemented, since it is more internally consistent.

Penny for any opinions further to Jason's and my own.

Here late, but I agree that "unaffected by local¹ context" is the best definition for isabs. No doubt it was discussions between Barney and myself that landed on that definition though, so maybe that opinion isn't worth a penny.

"Unaffected by abspath" is another good definition, though I believe we have that backwards because abspath uses isabs to check whether it needs to do anything.

My tendency on the kinds of situations distutils was in is to make paths absolute as early as possible, and keep track of what they should be relative to for cases where I need to output that. Since abspath is safe/consistent on all (path) inputs, and calculating relative paths from absolute paths should be consistent across versions and platforms, this seems an approach that will work everywhere.

A relative path is unaffected by moving it to another machine, while an absolute path is theoretically no longer valid, but I don't have a good name for this kind of context. ↩

Thanks for the greater context. My recommendation, unless there are other reports of use-cases adversely affected by this change, let's accept the new implementation and charge ahead.

Just wanted to point out that it might be a security risk that paths like / and \ return False on Windows as of Python 3.13, even though in Windows (at least Windows 10) both paths refer to the root of the drive. I often use os.path.isabs (along with a few other checks) to make sure user input doesn't refer to files outside the current working directory, but a bad actor could easily use this behavior to get around the checks.

I often use os.path.isabs (along with a few other checks) to make sure user input doesn't refer to files outside the current working directory, but a bad actor could easily use this behavior to get around the checks.

You need a better check anyway, to detect malicious paths like a\b\..\..\..\..\..\..\Windows\System32\.... I personally resolve the path and then try pathlib.Path.relative_to and handle the exception, but I'm sure there's an ntpath equivalent.

Flag absolute src paths again on Python 3.13

Navigation Menu

Repository navigation

Add os.path.isrelative() and improve ntpath.isabs() #44626

Linked PRs

26 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add os.path.isrelative() and improve ntpath.isabs() #44626

Description

Linked PRs

Activity

jongfoster commented on Feb 26, 2007

jongfoster commented on Feb 26, 2007

jorend commented on Mar 7, 2007

jorend commented on Mar 7, 2007

jongfoster commented on Mar 7, 2007

mhammond commented on Mar 8, 2007

jongfoster commented on Mar 8, 2007

tiran commented on Dec 18, 2007

BreamoreBoy commented on Jul 14, 2010

eckhardt commented on Sep 29, 2010

terryjreedy commented on Jun 12, 2011

26 remaining items

jaraco commented on Jan 4, 2025

barneygale commented on Jan 4, 2025

barneygale commented on Jan 4, 2025

jaraco commented on Jan 4, 2025

zooba commented on Feb 3, 2025

Footnotes

jaraco commented on Feb 15, 2025

the-can-of-soup commented on Feb 16, 2025

zooba commented on Feb 18, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions