Skip to content

bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) #7160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
giampaolo merged 114 commits into from
Jun 12, 2018
+595 −19
Merged

bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) #7160

Changes from 1 commit
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
1a72c01
have shutil.copyfileobj use sendfile() if possible
giampaolo May 22, 2018
77c4bfa
refactoring: use ctx manager
giampaolo May 22, 2018
2afa04a
add test with non-regular file obj
giampaolo May 22, 2018
542cd17
emulate case where file size can't be determined
giampaolo May 22, 2018
3520c6c
reference _copyfileobj_sendfile directly
giampaolo May 22, 2018
050a722
add test for offset() at certain position
giampaolo May 22, 2018
c1fd38a
add test for empty file
giampaolo May 22, 2018
2ab6317
add test for non regular file dst
giampaolo May 22, 2018
dacc3b6
small refactoring
giampaolo May 22, 2018
29d5881
leave copyfileobj() alone in order to not introduce any incompatibility
giampaolo May 24, 2018
114c4de
minor refactoring
giampaolo May 24, 2018
501c0dd
remove old test
giampaolo May 24, 2018
41b4506
update docstring
giampaolo May 24, 2018
fdb0973
update docstring; rename exception class
giampaolo May 24, 2018
64d2bc5
detect platforms which only support file to socket zero copy
giampaolo May 24, 2018
3a3c8ef
don't run test on platforms where file-to-file zero copy is not suppo…
giampaolo May 24, 2018
7861737
use tempfiles
giampaolo May 24, 2018
f3eecfd
reset verbosity
giampaolo May 24, 2018
f67ce57
add test for smaller chunks
giampaolo May 24, 2018
d457254
add big file size test
giampaolo May 24, 2018
8eb211d
add comment
giampaolo May 24, 2018
a0fe703
update doc
giampaolo May 24, 2018
7296147
update whatsnew doc
giampaolo May 24, 2018
d0c3bba
update doc
giampaolo May 24, 2018
2cafd80
catch Exception
giampaolo May 24, 2018
bb2a75f
remove unused import
giampaolo May 24, 2018
e5025dc
add test case for error on second sendfile() call
giampaolo May 24, 2018
a36a534
turn docstring into comment
giampaolo May 24, 2018
e9da3fa
add one more test
giampaolo May 24, 2018
9fcc2e7
update comment
giampaolo May 24, 2018
4f32242
add Misc/NEWS entry
giampaolo May 24, 2018
24ad25a
get rid of COPY_BUFSIZE; it belongs to another PR
giampaolo May 25, 2018
24d20e6
update doc
giampaolo May 25, 2018
7b6e576
expose posix._fcopyfile() for OSX
giampaolo May 27, 2018
b82ddc9
Merge branch 'master' into shutil-osx-copyfile
giampaolo May 27, 2018
b62b61e
merge from linux branch
giampaolo May 27, 2018
34e9618
merge from linux branch
giampaolo May 27, 2018
6b20902
expose fcopyfile
giampaolo May 27, 2018
abf3ecb
arg clinic for the win implementation
giampaolo May 28, 2018
91e492c
convert path type to path_t
giampaolo May 28, 2018
e02c69d
expose CopyFileW
giampaolo May 28, 2018
73837e2
fix windows tests
giampaolo May 28, 2018
28be4c1
release GIL
giampaolo May 28, 2018
6c59adf
minor refactoring
giampaolo May 28, 2018
700629d
update doc
giampaolo May 28, 2018
077912e
update comment
giampaolo May 28, 2018
62c6568
update docstrings
giampaolo May 28, 2018
a40a755
rename functions
giampaolo May 28, 2018
7ba0085
rename test classes
giampaolo May 28, 2018
6c96d97
update doc
giampaolo May 28, 2018
80fbe6e
update doc
giampaolo May 28, 2018
fdf4bcb
update docstrings and comments
giampaolo May 28, 2018
185f130
avoid do import nt|posix modules if unnecessary
giampaolo May 28, 2018
c8c98ae
set nt|posix modules to None if not available
giampaolo May 28, 2018
17bb5e6
micro speedup
giampaolo May 28, 2018
d8b9bf9
update description
giampaolo May 28, 2018
b59ac57
add doc note
giampaolo May 28, 2018
8eefce7
use better wording in doc
giampaolo May 29, 2018
4fc8c6b
Merge branch 'master' into shutil-zero-copy
giampaolo May 30, 2018
3048e3d
rename function using 'fastcopy' prefix instead of 'zerocopy'
giampaolo May 30, 2018
11102e1
use :ref: in rst doc
giampaolo May 30, 2018
7545273
change wording in doc
giampaolo May 30, 2018
3261b74
add test to make sure sendfile() doesn't get called aymore in case it…
giampaolo May 30, 2018
51c476d
move CopyFileW in _winapi and actually expose CopyFileExW instead
giampaolo May 30, 2018
729dd23
fix line endings
giampaolo May 30, 2018
1823828
add tests for mode bits
giampaolo May 30, 2018
a9d6a07
add docstring
giampaolo May 30, 2018
e3ce917
remove test file mode class; let's keep it for later when Istart addr…
giampaolo May 30, 2018
f81a0ec
update doc to reflect new changes
giampaolo May 30, 2018
3e7475b
update doc
giampaolo May 30, 2018
05dd3cf
adjust tests on win
giampaolo May 31, 2018
9b54930
fix argument clinic error
giampaolo May 31, 2018
2bec11c
update doc
giampaolo May 31, 2018
c87648f
OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags ar…
giampaolo May 31, 2018
941f740
osx / copyfile: use path_t instead of char
giampaolo May 31, 2018
4d28c12
do not set dst name in the OSError exception in order to remain consi…
giampaolo May 31, 2018
2149b8b
add same file test
giampaolo May 31, 2018
6a02a2a
add test for same file
giampaolo May 31, 2018
2287508
have osx copyfile() pre-emptively check if src and dst are the same, …
giampaolo May 31, 2018
b9da5d5
turn PermissionError into appropriate SameFileError
giampaolo May 31, 2018
c921f46
expose ERROR_SHARING_VIOLATION in order to raise more appropriate Sam…
giampaolo May 31, 2018
bb24490
honour follow_symlinks arg when using CopyFileEx
giampaolo May 31, 2018
fef8b32
update Misc/NEWS
giampaolo May 31, 2018
71be453
expose CreateDirectoryEx mock
giampaolo Jun 5, 2018
6035fe2
change C type
giampaolo Jun 6, 2018
8dc651e
CreateDirectoryExW actual implementation
giampaolo Jun 6, 2018
5d0eada
provide specific makedirs() implementation for win
giampaolo Jun 6, 2018
d67cdc5
Merge branch 'shutil-zero-copy-8' of https://github.com/giampaolo/cpy…
giampaolo Jun 6, 2018
f65c8ae
fix typo
giampaolo Jun 6, 2018
9c4508e
skeleton for SetNamedSecurityInfo
giampaolo Jun 6, 2018
bb1fee6
get security info for src path
giampaolo Jun 6, 2018
566898a
finally set security attrs
giampaolo Jun 6, 2018
f435053
add unit tests
giampaolo Jun 6, 2018
30c9a57
mimick os.makedirs() behavior and raise if dst dir exists
giampaolo Jun 6, 2018
33f362f
set 2 paths for OSError object
giampaolo Jun 6, 2018
e17e729
set 2 paths for OSError object
giampaolo Jun 6, 2018
bc46f75
expand windows test
giampaolo Jun 6, 2018
cabbc02
in case of exception on os.sendfile() set filename and filename2 exce…
giampaolo Jun 6, 2018
d22ee08
set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX
giampaolo Jun 6, 2018
7a08203
update doc
giampaolo Jun 7, 2018
ab284e9
do not use CreateDirectoryEx() in copytree() if source dir is a symli…
giampaolo Jun 7, 2018
ac9479d
use bytearray() and readinto()
giampaolo Jun 7, 2018
fd77a7e
use memoryview() with bytearray()
giampaolo Jun 7, 2018
42a597e
refactoring + introduce a new _fastcopy_binfileobj() fun
giampaolo Jun 8, 2018
5008a8d
remove CopyFileEx and other C wrappers
giampaolo Jun 8, 2018
e89dd20
remove code related to CopyFileEx
giampaolo Jun 8, 2018
c0dc4b8
Recognize binary files in copyfileobj()
giampaolo Jun 8, 2018
29b9730
set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable
giampaolo Jun 8, 2018
a1bed32
use ctx manager for memoryview()
giampaolo Jun 8, 2018
d9d27a7
update doc
giampaolo Jun 9, 2018
17bd78b
remove outdated doc
giampaolo Jun 9, 2018
b1d4917
remove last CopyFileEx remnants
giampaolo Jun 9, 2018
5ce94e4
OSX - use fcopyfile(3) instead of copyfile(3)
giampaolo Jun 12, 2018
07bcef5
update doc
giampaolo Jun 12, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update docstring
  • Loading branch information
@giampaolo
giampaolo committed May 24, 2018
commit 41b4506b3f637b3bb388e37daf08cee12156aec1
7 changes: 4 additions & 3 deletions Lib/shutil.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,10 +129,11 @@ def copyfileobj(fsrc, fdst, length=COPY_BUFSIZE):
def _copyfileobj2(fsrc, fdst):

@ncoghlan ncoghlan May 29, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest _copyfileobj_fast as the name for this.

@giampaolo giampaolo May 29, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about _zerocopy_fileobj in order to remain consistent with _zerocopy_osx, _zerocopy_win and _zerocopy_sendfile?

@ncoghlan ncoghlan May 30, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given @eryksun's comments, a more technically accurate set of names might be _fastcopy_osx, _fastcopy_win, _fastcopy_sendfile, _fastcopy_fileobj, and _GiveUpOnFastCopy.

@giampaolo giampaolo May 30, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. Done.

"""Same as above but tries to use zero-copy sendfile(2) syscall
(faster). This is used by copyfile(), copy() and copy2() in order
to leave copyfileobj() alone and not introduce backward
incompatibilities.
to leave copyfileobj() alone and not introduce any backward
incompatibility.
E.g. by using sendfile() fdst cannot be opened in "a"(ppend) mode
and its offset doesn't get updated.
and its offset doesn't get updated. Also, fsrc and fdst may be
opened in text mode.
"""
if _HAS_SENDFILE:

@ncoghlan ncoghlan May 29, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given a configure check for _HAS_SENDFILE, the initial check for both it and _HAS_FCOPYFILE can be moved up to the module level, and define _copyfileobj_fast differently based on the results. If both flags are false, then _copyfileobj_fast can be aliased directly to copyfileobj. (I believe Mac OS X defines both operations, so it will be important to give _HAS_FCOPYFILE precedence when deciding which implementation of _copyfileobj_fast to use)

The dynamic check to handle the case where a Python built on a newer distro is run against an older kernel (and hence still gets ENOTSOCK at runtime) can then also be handled by setting _copyfileobj_fast = copyfileobj rather than by setting a flag that needs to be checked on every call.

@giampaolo giampaolo May 30, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my comment above.

try:
Expand Down