Repository navigation
shutil.copyfile should use ReFS block-level linking on Windows (by invoking CopyFileW) #137074
Description
Activity
This was looked at in 2018 and decided against in gh-77852 which added the platform-specific optimizations because it was slower (see: #77852 (comment) high level, for details see: #7160 (comment) and the comments starting at #7160 (comment)).
In 2019 it looks like more discussion of CopyFileW occured (gh-83087) with the leaning "use CopyFile" everywhere. It sounds like in newer Windows this might be faster (so this should be gated by a windows version potentially), would need benchmarks to validate.
Reacted by Dicocopy2already uses the more efficient way, because the default copy API on Windows includes metadata.copyfileunfortunately is not supposed to copy metadata, and so it can't use the fast option.If we can change the definition of
copyfileto allow it to copy metadata without breaking anyone, then we can make it faster. Otherwise, we're stuck trying to imitate POSIX semantics.copy2 is affected by this issue
What does this mean? Are you saying that
copy2doesn't use copy-on-write? Or that it does?copy2 is affected by this issue
What does this mean? Are you saying that
copy2doesn't use copy-on-write? Or that it does?I am saying that copy2 does NOT use copy-on-write. I replaced copy2 with a call to CopyFileW via ctypes and it saves loads of space on ReFS drives with the change to CopyFileW.
That surprises me, they should both be calling through to the same underlying API with the same settings (based on how we're calling it from
_winapi).Can you share your test scripts and system configuration (OS version, in particular)?
It is Windows 11 version 26100, python 3.5.2 I think running a script that was converted by pyinstaller and invoking shutil.copy2.
When I looked at the code (linked in the issue) it appeared unimplemented, if you're positive that it is for copy2 then feel free to close.- I can see the part of copy2 that should cover this case (it appears to call CopyFile2 from winapi). I'll assume this is a python version thing and close the issue. I don't have a minimal test case, just many filled up hard drives, but we have a solution now with ctypes CopyFileW that works great.
Reacted by Steve Dower
Bug report
Bug description:
copyfile for copies within a single ReFS volume (or Dev Drive) on Windows 11 does not take advantage of block-level linking capability present in ReFS file system. On the other hand, the kernel automatically does so when you invoke CopyFileW these days. C#/.NET use kernel functions for file copy APIs and automatically take advantage of this too.
Python does not offer an API that takes advantage of block-level linking except for directly invoking CopyFileW using windlls.kernel32 from ctypes.
https://docs.python.org/3/library/shutil.html#shutil-platform-dependent-efficient-copy-operations
CopyFileW should be used instead of streaming data from one file to another, at least in circumstances where block-level linking is possible.
Relevant code:
cpython/Lib/shutil.py
Line 313 in e93c30d
CPython versions tested on:
3.9
Operating systems tested on:
Windows