Repository navigation
[easy doc] Document os.rename() behavior on Windows when src and dst are on different filesystems #72542
Description
| BPO | 28356 |
|---|---|
| Nosy | @pfmoore, @tjguk, @zware, @eryksun, @zooba, @rhyn0 |
| PRs | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2016-10-04.13:42:32.460>
labels = ['easy', '3.8', '3.9', '3.10', 'type-feature', 'OS-windows', 'docs']
title = '[easy doc] Document os.rename() behavior on Windows when src and dst are on different filesystems'
updated_at = <Date 2021-08-02.13:03:11.787>
user = 'https://bugs.python.org/stephan'bugs.python.org fields:
activity = <Date 2021-08-02.13:03:11.787>
actor = 'vstinner'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Windows']
creation = <Date 2016-10-04.13:42:32.460>
creator = 'stephan'
dependencies = []
files = ['50180']
hgrepos = []
issue_num = 28356
keywords = ['patch', 'easy', 'newcomer friendly']
message_count = 13.0
messages = ['278035', '278036', '278038', '278040', '278041', '278048', '278113', '387667', '388283', '388295', '388297', '388322', '398201']
nosy_count = 9.0
nosy_names = ['paul.moore', 'tim.golden', 'docs@python', 'zach.ware', 'eryksun', 'steve.dower', 'stephan', 'scienidlex', 'rhyn0']
pr_nums = ['27376']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue28356'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
Activity
stephan commentedon Oct 4, 2016
Hi,
I am just migrating my code from python 2.7.12 to 3.5.2
on Windows and stumbled on the following difference:
In python 2.7.12:
os.rename(filepath1, filepath2)
works even if
filepath1 and filepath2 are on different drives
(or one on a local drive, the other on a network share).
In python 3.5.2 I get for the same operation:
"OSError: [WinError 17] The system cannot move the file to a different disk drive"
My question:
I did find nothing, but I think it should be mentioned,
otherwise I assume it's a bug.
vstinner commentedon Oct 4, 2016
On Python 3 on Windows, os.rename() is implemented as MoveFileExW() with flags=0.
The doc says: "When moving a file, the destination can be on a different file system or volume. If the destination is on another drive, you must set the MOVEFILE_COPY_ALLOWED flag in dwFlags."
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365240%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396
I guess that the portable fix is to try rename() or fall back on copy(src, dst) + delete(src).
--
On Python 2 on Windows, os.rename() is implemented as MoveFileW(). It seems like this function behaves as MoveFileEx() called with MOVEFILE_COPY_ALLOWED:
"A new file may be on a different file system or drive."
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365239(v=vs.85).aspx
--
Should we add a flag to os.rename() to allow copy, to have a portable API?
[-]os.rename different in python 2.7.12 and python 3.5.2[/-][+]Windows: os.rename different in python 2.7.12 and python 3.5.2[/+]vstinner commentedon Oct 4, 2016
Ah, on Linux rename() also fails if src and dst are on two different filesystems: OSError: [Errno 18] Invalid cross-device link.
By the way, the Python document doesn't say anything about operation on two different filesystems :-/
https://docs.python.org/dev/library/os.html#os.rename
eryksun commentedon Oct 4, 2016
3.3 added os.replace, which on Windows entailed a switch from calling MoveFile to MoveFileEx in order to specify the MOVEFILE_REPLACE_EXISTING flag. However, not passing the MOVEFILE_COPY_ALLOWED broke compatibility with os.rename on Windows for versions prior to 3.3. I don't know whether or not this was discussed as an intentional breaking change in order to align the behavior with POSIX rename(). The change seems reasonable to me, plus at this point I don't think much can be done other than to add a note to the docs that the behavior changed in 3.3.
stephan commentedon Oct 4, 2016
Hi,
I tryed os.replace() as replacement for os.rename() too,
but as you said it does not work if the files are on different drives.
For now I switched to shutil.move() but I suppose its not
as performant/optimal as an "move" or "rename" directly supported
by the OS.
eryksun commentedon Oct 4, 2016
In scanning over bpo-8828, I see no discussion of the consequences of not using MOVEFILE_COPY_ALLOWED in Antoine's patch, so it appears that this behavior change was unintentional.
Correct, the copy employed by shutil.move for a cross-volume move is not as optimized as what MoveFile uses (basically CopyFile), but unless you're moving a file that's very large it shouldn't matter, and even then without testing I don't know if it's a significant difference relative to the throughput of the disk(s) involved.
vstinner commentedon Oct 5, 2016
Depending on your point of view, it *can* be seen as an enhancement.
See the documentation of MOVEFILE_COPY_ALLOWED: "If the file is
successfully copied to a different volume and the original file is
unable to be deleted, the function succeeds leaving the source file
intact." This behaviour can be seen as a source of bug. It can be
acceptable if if is expected.
Since shutil.move() already implements the copy+delete fallback, I
suggest to *not* change os.rename() but *document* the behaviour
change:
file systems
https://docs.python.org/dev/library/shutil.html#shutil.move already
explains well the behaviour when two different filesystems are used:
"... os.rename() is used. Otherwise, src is copied to dst using
copy_function and then remove ..."
I also suggest to mention in shutil.move() doc that the source can be
left undeleted if delete fails for some reason when copy+delete is
used.
eryksun commentedon Feb 25, 2021
The documentation of os.rename() should mention that on Windows the "operation will fail if src and dst are on different filesystems". For POSIX, instead of "will" it says "may", but the failure is a certainty in Windows since MOVEFILE_COPY_ALLOWED isn't used.
7 remaining items
[-]Windows: os.rename different in python 2.7.12 and python 3.5.2[/-][+][easy doc] Document os.rename() behavior on Windows when src and dst are on different filesystems[/+]vstinner commentedon Mar 8, 2021
This issue is now a matter of *documenting* the Windows behavior. Is there any volunteer to propose a PR?
scienidlex commentedon Mar 8, 2021
i have problem with python version 3.9.1
eryksun commentedon Mar 8, 2021
T-VEy, this is a documentation-only issue for versions of Python that are under active development for bug fixes and enhancements -- 3.8, 3.9, and 3.10.
scienidlex commentedon Mar 9, 2021
Eryk, can you please go on details on how are they using different strategies to fix bugs but still it's not enough for the time being?Because I know nothing about python and I recognized it in a minute of opening the files.I don't even have it installed on my pc.I don't even have python, not even version 2.7.
On Mon, Mar 8, 2021 at 23:48, Eryk Sun<report@bugs.python.org> wrote:
Eryk Sun <eryksun@gmail.com> added the comment:
T-VEy, this is a documentation-only issue for versions of Python that are under active development for bug fixes and enhancements -- 3.8, 3.9, and 3.10.
----------
versions: +Python 3.10, Python 3.8
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue28356\>
rhyn0 commentedon Jul 26, 2021
Hi all,
This is my first issue so feedback is welcome.
Following @vstinner 's suggestions:
And from @eryksun :
Just 2 short lines:
2292,6 +2292,8 @@ features:
will fail with an :exc:`OSError` subclass in a number of cases:
+ The operation will fail if *src* and *dst* are on different filesystems. Use
+ :func:`shutil.move` to support moves to a different filesystem.
If nothing to change, I will make a PR soon.
furkanonder commentedon May 11, 2023
@eryksun The problem seems to be solved. I think we can close the problem.
pythongh-72542: State that os.rename() always fails across filesystem…