Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions Doc/library/os.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2807,6 +2807,12 @@ features:
This function can also support :ref:`paths relative to directory descriptors
<dir_fd>`.

On MacOS and Linux, *path* can also be a file descriptor referring to a
symbolic link. In that case, *dir_fd* must be ``None``, and the return

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another sensible behaviour would be to ignore dir_fd completely, but I think raising an error here is the better alternative.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to document how to get such a file descriptor (O_SYMLINK and O_PATH|O_NOFOLLOW).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this to the docs.

value will be a ``bytes`` object.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This just so happened - maybe a str is more reasonable?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, readlins returns str (with surogatepass encoding, so you can get the bytes out losslessly).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated: The implementation returns a "string" (via PyUnicode_DecodeFSDefaultAndSize), the tests check for equality with a string ("symlink" rather than b"symlink"), and the docs are updated, too.

On other operating systems, a ``NotImplementedError`` is raised if *path*
is an integer.

When trying to resolve a path that may contain links, use
:func:`~os.path.realpath` to properly handle recursion and platform
differences.
Expand All @@ -2829,6 +2835,10 @@ features:
substitution path (which typically includes ``\\?\`` prefix) rather
than the optional "print name" field that was previously returned.

.. versionchanged:: 3.16
Comment thread
vstinner marked this conversation as resolved.
Outdated
Accepts file descriptors pointing to symbolic links as *path* on
Linux and MacOS.

.. function:: remove(path, *, dir_fd=None)

Remove (delete) the file *path*. If *path* is a directory, an
Expand Down
4 changes: 4 additions & 0 deletions Doc/whatsnew/3.16.rst
Original file line number Diff line number Diff line change
Expand Up @@ -509,6 +509,10 @@ os
now raises :exc:`PermissionError` instead of the functions being unavailable.
(Contributed by Md Arif in :gh:`152936`.)

* :func:`os.readlink` now accepts a file descriptor referring to a symlink on
Linux, Android and MacOS.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add Android to the other docs?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment thread
vstinner marked this conversation as resolved.
Outdated
(Contributed by OOTS in :gh:`157899`.)


pydoc
-----
Expand Down
72 changes: 72 additions & 0 deletions Lib/test/test_os/test_posix.py
Original file line number Diff line number Diff line change
Expand Up @@ -1881,6 +1881,67 @@ def test_readlink_dir_fd(self):
self.addCleanup(posix.unlink, fullname)
self.assertEqual(posix.readlink(name, dir_fd=dir_fd), 'symlink')


_support_readlink_with_fd = hasattr(os, 'readlink') and (
"HAVE_FREADLINK" in posix._have_functions # MacOS
Comment thread
vstinner marked this conversation as resolved.
Outdated
or (
os.readlink in os.supports_dir_fd
and sys.platform in ["linux", "android"]
)
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One open question is whether os.readlink() should be added to supports_fd. I don't have an opinion on that, happy to take others' lead here.

I'd say add them. O_SYMLINK/O_PATH|O_NOFOLLOW are the way to get fds referring to a symlink on their respective platform, so this can be just os.readlink in os.supports_dir_fd.
(Yes, I changed my mind after going through the code.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then, please also test that os.readlink in os.supports_dir_fd holds on MacOS, Linux & Android.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. os.readlink is now added automatically to os.supports_fd. I also added a test that checks presence/absence on all platforms where both os.readlink and os.supports_fd are defined.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_support_readlink_with_fd should now be equivalent to hasattr(os, 'readlink') and (os.readlink in os.supports_fd). Or more fancy, (getattr(os, 'readlink', None) in os.supports_fd)
The in expression can be used directly, as in the test above; no need for a _support_readlink_with_fd wariable.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm updating this, but I'd prefer to keep the _support_readlink_with_fd variable (just for avoiding code/logic duplication).

As a side effect, I'm now removing the self.assertEqual(os.readlink in os.supports_fd, self._support_readlink_with_fd) test that you asked for, since this test would be tautological. (It currently sits behind a @unittest.skipUnless(hasattr(os, "readlink")) guard.)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, rather than removing the test completely I added something closer to what you originally asked for:

    @unittest.skipUnless(sys.platform in ["linux", "android", "darwin", "ios"],
                         "feature not supported on this platform")
    def test_readlink_is_marked_as_supports_fd(self):
        self.assertIn(os.readlink, os.supports_fd)


def _open_symlink_as_fd(self, path):
open_flags = os.O_RDONLY
if hasattr(os, "O_SYMLINK"): # MacOS
open_flags |= os.O_SYMLINK
elif hasattr(os, "O_NOFOLLOW") and hasattr(os, "O_PATH"): # Linux
open_flags |= os.O_NOFOLLOW | os.O_PATH
else:
self.fail("lacking open flags for this test")
return os.open(path, open_flags)

@unittest.skipUnless(_support_readlink_with_fd,
"feature not supported on this platform")
def test_readlink_with_fd(self):
with self.prepare() as (dir_fd, name, fullname):
os.symlink("symlink", fullname)
self.addCleanup(posix.unlink, fullname)
fd = self._open_symlink_as_fd(fullname)
self.addCleanup(os.close, fd)
self.assertEqual(os.readlink(fd), b"symlink")

@unittest.skipUnless(_support_readlink_with_fd,
"feature not supported on this platform")
def test_readlink_with_fd_not_referring_to_symlink_throws(self):
with self.prepare_file() as (dir_fd, name, fullname):
fd = os.open(fullname, os.O_RDONLY)
self.addCleanup(os.close, fd)
with self.assertRaises(OSError):
Comment thread
encukou marked this conversation as resolved.
os.readlink(fd)

@unittest.skipUnless(_support_readlink_with_fd,
"feature not supported on this platform")
def test_readlink_with_fd_throws_if_both_fd_and_dir_fd_are_given(self):
with self.prepare() as (dir_fd, name, fullname):
os.symlink("symlink", fullname)
self.addCleanup(posix.unlink, fullname)
fd = self._open_symlink_as_fd(fullname)
self.addCleanup(os.close, fd)
with self.assertRaises(ValueError):
os.readlink(fd, dir_fd=dir_fd)

@unittest.skipIf(_support_readlink_with_fd,
"feature is supported on this platform")
def test_readlink_with_fd_throws_not_implemented_error(self):
# on unsupported platforms, we may not even be able to get a
# file descriptor for a symlink, so use a fd for an ordinary file
os_helper.create_empty_file(os_helper.TESTFN)
self.addCleanup(os_helper.unlink, os_helper.TESTFN)
fd = os.open(os_helper.TESTFN, os.O_RDONLY)
self.addCleanup(os.close, fd)
with self.assertRaises(NotImplementedError):
os.readlink(fd)
Comment thread
encukou marked this conversation as resolved.

@unittest.skipUnless(os.rename in os.supports_dir_fd, "test needs dir_fd support in os.rename()")
def test_rename_dir_fd(self):
with self.prepare_file() as (dir_fd, name, fullname), \
Expand Down Expand Up @@ -2620,6 +2681,17 @@ def test_readlink(self):
with self.assertRaisesRegex(NotImplementedError, "dir_fd unavailable"):
os.readlink("path", dir_fd=0)

def test_freadlink(self):
self._verify_available("HAVE_FREADLINK")
if self.mac_ver >= (13, 0):
self.assertIn("HAVE_FREADLINK", posix._have_functions)

else:
self.assertNotIn("HAVE_FREADLINK", posix._have_functions)

with self.assertRaisesRegex(NotImplementedError, "readlink cannot read file descriptors on this platform"):
Comment thread
vstinner marked this conversation as resolved.
Outdated
os.readlink(0)

def test_symlink(self):
self._verify_available("HAVE_SYMLINKAT")
if self.mac_ver >= (10, 10):
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
:func:`os.readlink` now accepts a file descriptor referring to a
symlink on Linux, Android and MacOS.
Comment thread
vstinner marked this conversation as resolved.
Outdated
18 changes: 15 additions & 3 deletions Modules/clinic/posixmodule.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

78 changes: 75 additions & 3 deletions Modules/posixmodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -505,6 +505,7 @@ static const unsigned int _Py_STATX_KNOWN = (STATX_BASIC_STATS | STATX_BTIME
# define HAVE_UNLINKAT_RUNTIME __builtin_available(macOS 10.10, iOS 8.0, *)
# define HAVE_OPENAT_RUNTIME __builtin_available(macOS 10.10, iOS 8.0, *)
# define HAVE_READLINKAT_RUNTIME __builtin_available(macOS 10.10, iOS 8.0, *)
# define HAVE_FREADLINK_RUNTIME __builtin_available(macOS 13.0, *)
# define HAVE_SYMLINKAT_RUNTIME __builtin_available(macOS 10.10, iOS 8.0, *)
# define HAVE_FUTIMENS_RUNTIME __builtin_available(macOS 10.13, iOS 11.0, tvOS 11.0, watchOS 4.0, *)
# define HAVE_UTIMENSAT_RUNTIME __builtin_available(macOS 10.13, iOS 11.0, tvOS 11.0, watchOS 4.0, *)
Expand Down Expand Up @@ -571,6 +572,10 @@ static const unsigned int _Py_STATX_KNOWN = (STATX_BASIC_STATS | STATX_BTIME
# define HAVE_READLINKAT_RUNTIME (readlinkat != NULL)
# endif

# ifdef _Py_HAVE_FREADLINK
# define HAVE_FREADLINK_RUNTIME (freadlink != NULL)
# endif

# ifdef HAVE_SYMLINKAT
# define HAVE_SYMLINKAT_RUNTIME (symlinkat != NULL)
# endif
Expand Down Expand Up @@ -10990,7 +10995,7 @@ os_unshare_impl(PyObject *module, int flags)
/*[clinic input]
os.readlink

path: path_t
path: path_t(allow_fd=True)
*
dir_fd: dir_fd(requires='readlinkat') = None

Expand All @@ -11002,22 +11007,65 @@ that directory.

dir_fd may not be implemented on your platform. If it is unavailable,
using it will raise a NotImplementedError.

On Linux, Android and MacOS, path may be a file descriptor referring to
Comment thread
vstinner marked this conversation as resolved.
Outdated
a symlink. If it is, dir_fd must be None, and the return value will be a
bytes object. (File descriptors for symlinks can be obtained with

os.open(..., os.O_RDONLY | os.O_PATH | os.O_NOFOLLOW)

on Linux and Android, and

os.open(..., os.O_RDONLY | os.O_SYMLINK)

on MacOS.)
[clinic start generated code]*/

static PyObject *
os_readlink_impl(PyObject *module, path_t *path, int dir_fd)
/*[clinic end generated code: output=d21b732a2e814030 input=03d10130870dbca8]*/
/*[clinic end generated code: output=d21b732a2e814030 input=30272a2c5fba427c]*/
{
#if defined(HAVE_READLINK)
char buffer[MAXPATHLEN+1];
ssize_t length;
#ifdef HAVE_READLINKAT
int readlinkat_unavailable = 0;
#endif
#ifdef _Py_HAVE_FREADLINK
int freadlink_unavailable = 0;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICS, we don't need the *_unavailable variables. The pattern is sometimes used because we can't raise exceptions inside Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS, but here, moving the Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to tightly wrap each freadlink/readlinkat/readlink call would allow setting the exception & returning direclty in the else blocks.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I dropped the freadlink_unavailable variable and the error is now raised directly in the else branch.

I also "inlined" all the invocations of Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS, each syscall (freadlink, readlinkat in the path->is_fd case, readlinkat in the old dir_fd case, readlink) are now directly surrounded by those macros.

This is technically a bit of a refactor and slightly out-of-scope of the issue - but I hope it's okay to piggy-back this refactor onto this MR.

#endif

if (path_and_dir_fd_invalid("readlink", path, dir_fd)) {
Comment thread
encukou marked this conversation as resolved.
return NULL;
}

if (path->is_fd) {
#if defined(__APPLE__) && defined(_Py_HAVE_FREADLINK)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we only want _Py_HAVE_FREADLINK here -- if any other platform has it, it's the function to call.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the defined(__APPLE__) check.

/* nop, freadlink is called below */
#elif defined(__linux__) && defined(HAVE_READLINKAT)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be useful to make this a configure variable, something like _Py_READLINKAT_SUPPORTS_EMPTY_PATH, only set on Linux.
Then, when someone ports CPython to another system that uses this (or to ancient Linux that doesn't), they can just change pyconfig.h or the configure script.

Alternately, use defined(HAVE_READLINKAT) && defined(O_PATH). O_PATH and empty pathname support were added in Linux 2.6.39; they look like it's essentially a single feature.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried with defined(HAVE_READLINKAT) && defined(O_PATH), but that broke on the emscripten build. (Apparently both Macros are defined there even though readlink is not supported on that platform AFAIK.)

So now _Py_READLINKAT_SUPPORTS_EMPTY_PATH is set automatically on Linux and Android (that have O_PATH) above. Anyone who really wants it can still force that variable.

I didn't add anything about this to the autoconfig part - I hope that's okay. If you want it threaded through/into the configure.ac somehow, I can look into that.

// Linux: readlinkat(dir_fd, "", ...) reads the symbolic link
// pointed to by dir_fd
dir_fd = path->fd;
path->narrow = "";

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Putting the values in these variables is misleading.
Instead, could you add a separate block below with a call to readlinkat(path->fd, "", ...) & the error handling?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. We now have a if (path->is_fd) { ... } else if (path->dir_fd != DEFAULT_DIR_FD) { ... } else { ... } structure.

The different implementations of this feature (freadlink on MacOS, readlinkat on Linux/Android, NotImplementedError on Android) are now all inside the if (path->is_fd) { ... } branch.

#else
PyErr_SetString(PyExc_NotImplementedError,
"readlink cannot read file descriptors on this platform");
return NULL;
Comment thread
encukou marked this conversation as resolved.
Outdated
#endif
}

Py_BEGIN_ALLOW_THREADS
#ifdef _Py_HAVE_FREADLINK
if (path->is_fd) {
if (HAVE_FREADLINK_RUNTIME) {
length = freadlink(path->fd, buffer, MAXPATHLEN);
} else {
freadlink_unavailable = 1;
}
} else
#endif
#ifdef HAVE_READLINKAT
if (dir_fd != DEFAULT_DIR_FD) {
if (dir_fd != DEFAULT_DIR_FD || path->is_fd) {
if (HAVE_READLINKAT_RUNTIME) {
length = readlinkat(dir_fd, path->narrow, buffer, MAXPATHLEN);
} else {
Expand All @@ -11028,6 +11076,16 @@ os_readlink_impl(PyObject *module, path_t *path, int dir_fd)
length = readlink(path->narrow, buffer, MAXPATHLEN);
Py_END_ALLOW_THREADS


#ifdef _Py_HAVE_FREADLINK
if (freadlink_unavailable) {
PyErr_Format(PyExc_NotImplementedError,
"readlink cannot read file descriptors on this platform, "
"freadlink() is unavailable");
return NULL;
}
#endif

#ifdef HAVE_READLINKAT
if (readlinkat_unavailable) {
argument_unavailable_error(NULL, "dir_fd");
Expand All @@ -11052,6 +11110,12 @@ os_readlink_impl(PyObject *module, path_t *path, int dir_fd)
_Py_REPARSE_DATA_BUFFER *rdb = (_Py_REPARSE_DATA_BUFFER *)target_buffer;
PyObject *result = NULL;

if (path->is_fd) {
PyErr_SetString(PyExc_NotImplementedError,
"readlink cannot read file descriptors on this platform");
return NULL;
}

/* First get a handle to the reparse point */
Py_BEGIN_ALLOW_THREADS
reparse_point_handle = CreateFileW(
Expand Down Expand Up @@ -18881,6 +18945,10 @@ PROBE(probe_openat, HAVE_OPENAT_RUNTIME)
PROBE(probe_readlinkat, HAVE_READLINKAT_RUNTIME)
#endif

#ifdef _Py_HAVE_FREADLINK
PROBE(probe_freadlink, HAVE_FREADLINK_RUNTIME)
#endif

#ifdef HAVE_SYMLINKAT
PROBE(probe_symlinkat, HAVE_SYMLINKAT_RUNTIME)
#endif
Expand Down Expand Up @@ -18948,6 +19016,10 @@ static const struct have_function {
{ "HAVE_FPATHCONF", NULL },
#endif

#ifdef _Py_HAVE_FREADLINK
{ "HAVE_FREADLINK", probe_freadlink },
#endif

#ifdef HAVE_FSTATAT
{ "HAVE_FSTATAT", probe_fstatat },
#endif
Expand Down
42 changes: 42 additions & 0 deletions configure

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions configure.ac
Original file line number Diff line number Diff line change
Expand Up @@ -5568,6 +5568,7 @@ fi
# raise an error if used at runtime. Force these symbols off.
if test "$ac_sys_system" != "iOS" ; then
AC_CHECK_FUNCS([dup3 getentropy getgroups pipe2 system])
PY_CHECK_FUNC_PRIVATE([freadlink], [@%:@include <unistd.h>])
fi

AC_CHECK_DECL([dirfd],
Expand Down
3 changes: 3 additions & 0 deletions pyconfig.h.in
Original file line number Diff line number Diff line change
Expand Up @@ -2201,6 +2201,9 @@
/* Defined if _Float16 C type is supported */
#undef _Py_HAVE_FLOAT16

/* Define if you have the 'freadlink' function. */
#undef _Py_HAVE_FREADLINK

/* Define if you have a working iconv() function. */
#undef _Py_HAVE_ICONV

Expand Down
Loading