Repository navigation
gh-157899: Add support for fds pointing to symlinks in os.readlink(). #157915
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 3 commits
2cbd30e
8bb6f09
8196b59
f96963a
7b566b7
253c3b4
20fa1e2
6c8f5f1
9bf9236
c8d85a6
83fcbf4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2807,6 +2807,12 @@ features: | |
| This function can also support :ref:`paths relative to directory descriptors | ||
| <dir_fd>`. | ||
|
|
||
| On MacOS and Linux, *path* can also be a file descriptor referring to a | ||
| symbolic link. In that case, *dir_fd* must be ``None``, and the return | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It would be nice to document how to get such a file descriptor (
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added this to the docs. |
||
| value will be a ``bytes`` object. | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This just so happened - maybe a
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes,
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Updated: The implementation returns a "string" (via |
||
| On other operating systems, a ``NotImplementedError`` is raised if *path* | ||
| is an integer. | ||
|
|
||
| When trying to resolve a path that may contain links, use | ||
| :func:`~os.path.realpath` to properly handle recursion and platform | ||
| differences. | ||
|
|
@@ -2829,6 +2835,10 @@ features: | |
| substitution path (which typically includes ``\\?\`` prefix) rather | ||
| than the optional "print name" field that was previously returned. | ||
|
|
||
| .. versionchanged:: 3.16 | ||
|
vstinner marked this conversation as resolved.
Outdated
|
||
| Accepts file descriptors pointing to symbolic links as *path* on | ||
| Linux and MacOS. | ||
|
|
||
| .. function:: remove(path, *, dir_fd=None) | ||
|
|
||
| Remove (delete) the file *path*. If *path* is a directory, an | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -509,6 +509,10 @@ os | |
| now raises :exc:`PermissionError` instead of the functions being unavailable. | ||
| (Contributed by Md Arif in :gh:`152936`.) | ||
|
|
||
| * :func:`os.readlink` now accepts a file descriptor referring to a symlink on | ||
| Linux, Android and MacOS. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you add Android to the other docs?
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done.
vstinner marked this conversation as resolved.
Outdated
|
||
| (Contributed by OOTS in :gh:`157899`.) | ||
|
|
||
|
|
||
| pydoc | ||
| ----- | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1881,6 +1881,67 @@ def test_readlink_dir_fd(self): | |
| self.addCleanup(posix.unlink, fullname) | ||
| self.assertEqual(posix.readlink(name, dir_fd=dir_fd), 'symlink') | ||
|
|
||
|
|
||
| _support_readlink_with_fd = hasattr(os, 'readlink') and ( | ||
| "HAVE_FREADLINK" in posix._have_functions # MacOS | ||
|
vstinner marked this conversation as resolved.
Outdated
|
||
| or ( | ||
| os.readlink in os.supports_dir_fd | ||
| and sys.platform in ["linux", "android"] | ||
| ) | ||
| ) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I'd say add them.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Then, please also test that
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm updating this, but I'd prefer to keep the As a side effect, I'm now removing the
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Okay, rather than removing the test completely I added something closer to what you originally asked for: @unittest.skipUnless(sys.platform in ["linux", "android", "darwin", "ios"],
"feature not supported on this platform")
def test_readlink_is_marked_as_supports_fd(self):
self.assertIn(os.readlink, os.supports_fd) |
||
|
|
||
| def _open_symlink_as_fd(self, path): | ||
| open_flags = os.O_RDONLY | ||
| if hasattr(os, "O_SYMLINK"): # MacOS | ||
| open_flags |= os.O_SYMLINK | ||
| elif hasattr(os, "O_NOFOLLOW") and hasattr(os, "O_PATH"): # Linux | ||
| open_flags |= os.O_NOFOLLOW | os.O_PATH | ||
| else: | ||
| self.fail("lacking open flags for this test") | ||
| return os.open(path, open_flags) | ||
|
|
||
| @unittest.skipUnless(_support_readlink_with_fd, | ||
| "feature not supported on this platform") | ||
| def test_readlink_with_fd(self): | ||
| with self.prepare() as (dir_fd, name, fullname): | ||
| os.symlink("symlink", fullname) | ||
| self.addCleanup(posix.unlink, fullname) | ||
| fd = self._open_symlink_as_fd(fullname) | ||
| self.addCleanup(os.close, fd) | ||
| self.assertEqual(os.readlink(fd), b"symlink") | ||
|
|
||
| @unittest.skipUnless(_support_readlink_with_fd, | ||
| "feature not supported on this platform") | ||
| def test_readlink_with_fd_not_referring_to_symlink_throws(self): | ||
| with self.prepare_file() as (dir_fd, name, fullname): | ||
| fd = os.open(fullname, os.O_RDONLY) | ||
| self.addCleanup(os.close, fd) | ||
| with self.assertRaises(OSError): | ||
|
encukou marked this conversation as resolved.
|
||
| os.readlink(fd) | ||
|
|
||
| @unittest.skipUnless(_support_readlink_with_fd, | ||
| "feature not supported on this platform") | ||
| def test_readlink_with_fd_throws_if_both_fd_and_dir_fd_are_given(self): | ||
| with self.prepare() as (dir_fd, name, fullname): | ||
| os.symlink("symlink", fullname) | ||
| self.addCleanup(posix.unlink, fullname) | ||
| fd = self._open_symlink_as_fd(fullname) | ||
| self.addCleanup(os.close, fd) | ||
| with self.assertRaises(ValueError): | ||
| os.readlink(fd, dir_fd=dir_fd) | ||
|
|
||
| @unittest.skipIf(_support_readlink_with_fd, | ||
| "feature is supported on this platform") | ||
| def test_readlink_with_fd_throws_not_implemented_error(self): | ||
| # on unsupported platforms, we may not even be able to get a | ||
| # file descriptor for a symlink, so use a fd for an ordinary file | ||
| os_helper.create_empty_file(os_helper.TESTFN) | ||
| self.addCleanup(os_helper.unlink, os_helper.TESTFN) | ||
| fd = os.open(os_helper.TESTFN, os.O_RDONLY) | ||
| self.addCleanup(os.close, fd) | ||
| with self.assertRaises(NotImplementedError): | ||
| os.readlink(fd) | ||
|
encukou marked this conversation as resolved.
|
||
|
|
||
| @unittest.skipUnless(os.rename in os.supports_dir_fd, "test needs dir_fd support in os.rename()") | ||
| def test_rename_dir_fd(self): | ||
| with self.prepare_file() as (dir_fd, name, fullname), \ | ||
|
|
@@ -2620,6 +2681,17 @@ def test_readlink(self): | |
| with self.assertRaisesRegex(NotImplementedError, "dir_fd unavailable"): | ||
| os.readlink("path", dir_fd=0) | ||
|
|
||
| def test_freadlink(self): | ||
| self._verify_available("HAVE_FREADLINK") | ||
| if self.mac_ver >= (13, 0): | ||
| self.assertIn("HAVE_FREADLINK", posix._have_functions) | ||
|
|
||
| else: | ||
| self.assertNotIn("HAVE_FREADLINK", posix._have_functions) | ||
|
|
||
| with self.assertRaisesRegex(NotImplementedError, "readlink cannot read file descriptors on this platform"): | ||
|
vstinner marked this conversation as resolved.
Outdated
|
||
| os.readlink(0) | ||
|
|
||
| def test_symlink(self): | ||
| self._verify_available("HAVE_SYMLINKAT") | ||
| if self.mac_ver >= (10, 10): | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| :func:`os.readlink` now accepts a file descriptor referring to a | ||
| symlink on Linux, Android and MacOS. | ||
|
vstinner marked this conversation as resolved.
Outdated
|
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -505,6 +505,7 @@ static const unsigned int _Py_STATX_KNOWN = (STATX_BASIC_STATS | STATX_BTIME | |
| # define HAVE_UNLINKAT_RUNTIME __builtin_available(macOS 10.10, iOS 8.0, *) | ||
| # define HAVE_OPENAT_RUNTIME __builtin_available(macOS 10.10, iOS 8.0, *) | ||
| # define HAVE_READLINKAT_RUNTIME __builtin_available(macOS 10.10, iOS 8.0, *) | ||
| # define HAVE_FREADLINK_RUNTIME __builtin_available(macOS 13.0, *) | ||
| # define HAVE_SYMLINKAT_RUNTIME __builtin_available(macOS 10.10, iOS 8.0, *) | ||
| # define HAVE_FUTIMENS_RUNTIME __builtin_available(macOS 10.13, iOS 11.0, tvOS 11.0, watchOS 4.0, *) | ||
| # define HAVE_UTIMENSAT_RUNTIME __builtin_available(macOS 10.13, iOS 11.0, tvOS 11.0, watchOS 4.0, *) | ||
|
|
@@ -571,6 +572,10 @@ static const unsigned int _Py_STATX_KNOWN = (STATX_BASIC_STATS | STATX_BTIME | |
| # define HAVE_READLINKAT_RUNTIME (readlinkat != NULL) | ||
| # endif | ||
|
|
||
| # ifdef _Py_HAVE_FREADLINK | ||
| # define HAVE_FREADLINK_RUNTIME (freadlink != NULL) | ||
| # endif | ||
|
|
||
| # ifdef HAVE_SYMLINKAT | ||
| # define HAVE_SYMLINKAT_RUNTIME (symlinkat != NULL) | ||
| # endif | ||
|
|
@@ -10990,7 +10995,7 @@ os_unshare_impl(PyObject *module, int flags) | |
| /*[clinic input] | ||
| os.readlink | ||
|
|
||
| path: path_t | ||
| path: path_t(allow_fd=True) | ||
| * | ||
| dir_fd: dir_fd(requires='readlinkat') = None | ||
|
|
||
|
|
@@ -11002,22 +11007,65 @@ that directory. | |
|
|
||
| dir_fd may not be implemented on your platform. If it is unavailable, | ||
| using it will raise a NotImplementedError. | ||
|
|
||
| On Linux, Android and MacOS, path may be a file descriptor referring to | ||
|
vstinner marked this conversation as resolved.
Outdated
|
||
| a symlink. If it is, dir_fd must be None, and the return value will be a | ||
| bytes object. (File descriptors for symlinks can be obtained with | ||
|
|
||
| os.open(..., os.O_RDONLY | os.O_PATH | os.O_NOFOLLOW) | ||
|
|
||
| on Linux and Android, and | ||
|
|
||
| os.open(..., os.O_RDONLY | os.O_SYMLINK) | ||
|
|
||
| on MacOS.) | ||
| [clinic start generated code]*/ | ||
|
|
||
| static PyObject * | ||
| os_readlink_impl(PyObject *module, path_t *path, int dir_fd) | ||
| /*[clinic end generated code: output=d21b732a2e814030 input=03d10130870dbca8]*/ | ||
| /*[clinic end generated code: output=d21b732a2e814030 input=30272a2c5fba427c]*/ | ||
| { | ||
| #if defined(HAVE_READLINK) | ||
| char buffer[MAXPATHLEN+1]; | ||
| ssize_t length; | ||
| #ifdef HAVE_READLINKAT | ||
| int readlinkat_unavailable = 0; | ||
| #endif | ||
| #ifdef _Py_HAVE_FREADLINK | ||
| int freadlink_unavailable = 0; | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. AFAICS, we don't need the
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. I dropped the I also "inlined" all the invocations of This is technically a bit of a refactor and slightly out-of-scope of the issue - but I hope it's okay to piggy-back this refactor onto this MR. |
||
| #endif | ||
|
|
||
| if (path_and_dir_fd_invalid("readlink", path, dir_fd)) { | ||
|
encukou marked this conversation as resolved.
|
||
| return NULL; | ||
| } | ||
|
|
||
| if (path->is_fd) { | ||
| #if defined(__APPLE__) && defined(_Py_HAVE_FREADLINK) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we only want
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Removed the |
||
| /* nop, freadlink is called below */ | ||
| #elif defined(__linux__) && defined(HAVE_READLINKAT) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it would be useful to make this a configure variable, something like Alternately, use
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I tried with So now I didn't add anything about this to the autoconfig part - I hope that's okay. If you want it threaded through/into the |
||
| // Linux: readlinkat(dir_fd, "", ...) reads the symbolic link | ||
| // pointed to by dir_fd | ||
| dir_fd = path->fd; | ||
| path->narrow = ""; | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Putting the values in these variables is misleading.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. We now have a The different implementations of this feature ( |
||
| #else | ||
| PyErr_SetString(PyExc_NotImplementedError, | ||
| "readlink cannot read file descriptors on this platform"); | ||
| return NULL; | ||
|
encukou marked this conversation as resolved.
Outdated
|
||
| #endif | ||
| } | ||
|
|
||
| Py_BEGIN_ALLOW_THREADS | ||
| #ifdef _Py_HAVE_FREADLINK | ||
| if (path->is_fd) { | ||
| if (HAVE_FREADLINK_RUNTIME) { | ||
| length = freadlink(path->fd, buffer, MAXPATHLEN); | ||
| } else { | ||
| freadlink_unavailable = 1; | ||
| } | ||
| } else | ||
| #endif | ||
| #ifdef HAVE_READLINKAT | ||
| if (dir_fd != DEFAULT_DIR_FD) { | ||
| if (dir_fd != DEFAULT_DIR_FD || path->is_fd) { | ||
| if (HAVE_READLINKAT_RUNTIME) { | ||
| length = readlinkat(dir_fd, path->narrow, buffer, MAXPATHLEN); | ||
| } else { | ||
|
|
@@ -11028,6 +11076,16 @@ os_readlink_impl(PyObject *module, path_t *path, int dir_fd) | |
| length = readlink(path->narrow, buffer, MAXPATHLEN); | ||
| Py_END_ALLOW_THREADS | ||
|
|
||
|
|
||
| #ifdef _Py_HAVE_FREADLINK | ||
| if (freadlink_unavailable) { | ||
| PyErr_Format(PyExc_NotImplementedError, | ||
| "readlink cannot read file descriptors on this platform, " | ||
| "freadlink() is unavailable"); | ||
| return NULL; | ||
| } | ||
| #endif | ||
|
|
||
| #ifdef HAVE_READLINKAT | ||
| if (readlinkat_unavailable) { | ||
| argument_unavailable_error(NULL, "dir_fd"); | ||
|
|
@@ -11052,6 +11110,12 @@ os_readlink_impl(PyObject *module, path_t *path, int dir_fd) | |
| _Py_REPARSE_DATA_BUFFER *rdb = (_Py_REPARSE_DATA_BUFFER *)target_buffer; | ||
| PyObject *result = NULL; | ||
|
|
||
| if (path->is_fd) { | ||
| PyErr_SetString(PyExc_NotImplementedError, | ||
| "readlink cannot read file descriptors on this platform"); | ||
| return NULL; | ||
| } | ||
|
|
||
| /* First get a handle to the reparse point */ | ||
| Py_BEGIN_ALLOW_THREADS | ||
| reparse_point_handle = CreateFileW( | ||
|
|
@@ -18881,6 +18945,10 @@ PROBE(probe_openat, HAVE_OPENAT_RUNTIME) | |
| PROBE(probe_readlinkat, HAVE_READLINKAT_RUNTIME) | ||
| #endif | ||
|
|
||
| #ifdef _Py_HAVE_FREADLINK | ||
| PROBE(probe_freadlink, HAVE_FREADLINK_RUNTIME) | ||
| #endif | ||
|
|
||
| #ifdef HAVE_SYMLINKAT | ||
| PROBE(probe_symlinkat, HAVE_SYMLINKAT_RUNTIME) | ||
| #endif | ||
|
|
@@ -18948,6 +19016,10 @@ static const struct have_function { | |
| { "HAVE_FPATHCONF", NULL }, | ||
| #endif | ||
|
|
||
| #ifdef _Py_HAVE_FREADLINK | ||
| { "HAVE_FREADLINK", probe_freadlink }, | ||
| #endif | ||
|
|
||
| #ifdef HAVE_FSTATAT | ||
| { "HAVE_FSTATAT", probe_fstatat }, | ||
| #endif | ||
|
|
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another sensible behaviour would be to ignore
dir_fdcompletely, but I think raising an error here is the better alternative.