Repository navigation
Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) #77852
Description
Activity
Patch in attachment uses platform specific zero-copy syscalls on Linux and Solaris (os.sendfile(2)), Windows (CopyFileW) and OSX (fcopyfile(2)) speeding up shutil.copyfile() and other functions using it (copy(), copy2(), copytree(), move()).
Average speedup for a 512MB file copy is +24% on Linux, +50% on OSX and +48% on Windows by copying file on the same partition (SSD disk was used).
Follows some benchmarks.
Setup
=====Create 128K, 8M, 512M file:
$ python -c "import os; f = open('f1', 'wb'); f.write(os.urandom(128 * 1024))" $ python -c "import os; f = open('f1', 'wb'); f.write(os.urandom(8 * 1024 * 1024))" $ python -c "import os; f = open('f1', 'wb'); f.write(os.urandom(512 * 1024 * 1024))"
Benchmark:
$ time ./python -m timeit -s 'import shutil; p1 = "f1"; p2 = "f2"' 'shutil.copyfile(p1, p2)'
Linux
=====128K copy (+13%):
without patch: 1000 loops, best of 5: 228 usec per loop real 0m1.756s user 0m0.386s sys 0m1.116swith patch: 1000 loops, best of 5: 198 usec per loop real 0m1.464s user 0m0.281s sys 0m0.958s
8MB copy (+24%):
without patch: 50 loops, best of 5: 10.1 msec per loop real 0m2.703s user 0m0.316s sys 0m1.847swith patch: 50 loops, best of 5: 7.78 msec per loop real 0m2.447s user 0m0.086s sys 0m1.682s
512MB copy (+26%):
without patch: 1 loop, best of 5: 872 msec per loop real 0m5.574s user 0m0.402s sys 0m3.115swith patch: 1 loop, best of 5: 646 msec per loop real 0m5.475s user 0m0.037s sys 0m2.959s
OSX
===128K copy (+8.5%):
without patch: 500 loops, best of 5: 508 usec per loop real 0m2.971s user 0m0.442s sys 0m2.168swith patch: 500 loops, best of 5: 464 usec per loop real 0m2.798s user 0m0.379s sys 0m2.031s
8MB copy (+67%):
without patch: 20 loops, best of 5: 32.8 msec per loop real 0m3.672s user 0m0.357s sys 0m1.434swith patch: 20 loops, best of 5: 10.8 msec per loop real 0m1.860s user 0m0.079s sys 0m0.719s
512MB copy (+50%):
without patch: 1 loop, best of 5: 953 msec per loop real 0m5.930s user 0m1.021s sys 0m4.835s with patch: 1 loop, best of 5: 480 msec per loop real 0m3.150s user 0m0.067s sys 0m2.740sWindows
=======128K copy (+69%):
without patch: 50 loops, best of 5: 6.45 msec per loop with patch: 50 loops, best of 5: 1.99 msec per loop8M copy (+64%):
without patch: 10 loops, best of 5: 22.6 msec per loop with patch: 50 loops, best of 5: 7.95 msec per loop512M copy (+48%):
without patch: 1 loop, best of 5: 1.21 sec per loop with patch: 1 loop, best of 5: 629 msec per loop- changed the title
[-]Efficient efficient zero-copy syscalls for shutil.copy* functions (Linux, OSX and Win)[/-][+]Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)[/+]on May 28, 2018 Nice, I really like this.
Apart from the usual bit of minor style issues, I couldn't see anything inherently wrong with the PR, but I'll leave the detailed reviews to those who'd have to maintain the code in the future. :)
Regarding the benchmarks, just to be sure, did you try reversing the run order to make sure you don't get unfair caching effects for the later runs?
socketpair commented
on May 28, 2018 socketpairmannequinMannequinhttp://man7.org/linux/man-pages/man2/ioctl_ficlonerange.2.html
That possibly should be used under Linux in order to really acheive zero-copying. Just like modern cp command.
Yes, I tried changing benchmarks order and zero-copy variants are always faster. As for instantaneous CoW copy, it is debatable. E.g. "cp" command does not do it by default:
https://unix.stackexchange.com/questions/80351/why-is-cp-reflink-auto-not-the-default-behaviour
I think shutil should follow the same lead, and perhaps provide a cow=bool argument in the future.For future reference, as per #7160 discussion, we decided not to use CopyFileEx on Windows and instead increase read() buffer size from 16KB to 1MB (Windows only) resulting in a 40.8% speedup (instead of 48%). Also copyfileobj() has been optimized on all platforms by using readinto()/memoryview()/bytearray().
Updated benchmarks on Windows:128KB copy (+27%)
without patch: 50 loops, best of 5: 7.69 sec per loop with patch: 50 loops, best of 5: 5.61 sec per loop8MB copy (+45.6%)
without patch: 10 loops, best of 5: 20.8 sec per loop with patch: 20 loops, best of 5: 11.3 sec per loop512MB copy (+40.8%)
without patch: 1 loop, best of 5: 1.26 sec per loop with patch: 1 loop, best of 5: 646 msec per loopStyXman commented
on Jun 13, 2018 StyXmanmannequinMannequinThanks Gianpaolo for pushing for this. Great job.
Thanks Gianpaolo for pushing for this. Great job.
I concur: great job! Cool optimization.
shutil.COPY_BUFSIZE isn't documented. Is it a deliberate choice?
https://docs.python.org/dev/library/shutil.html
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: