Repository navigation
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) #7160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+595
−19
Merged
Changes from 1 commit
Commits
Show all changes
114 commits
Select commit
Hold shift + click to select a range
1a72c01
have shutil.copyfileobj use sendfile() if possible
giampaolo 77c4bfa
refactoring: use ctx manager
giampaolo 2afa04a
add test with non-regular file obj
giampaolo 542cd17
emulate case where file size can't be determined
giampaolo 3520c6c
reference _copyfileobj_sendfile directly
giampaolo 050a722
add test for offset() at certain position
giampaolo c1fd38a
add test for empty file
giampaolo 2ab6317
add test for non regular file dst
giampaolo dacc3b6
small refactoring
giampaolo 29d5881
leave copyfileobj() alone in order to not introduce any incompatibility
giampaolo 114c4de
minor refactoring
giampaolo 501c0dd
remove old test
giampaolo 41b4506
update docstring
giampaolo fdb0973
update docstring; rename exception class
giampaolo 64d2bc5
detect platforms which only support file to socket zero copy
giampaolo 3a3c8ef
don't run test on platforms where file-to-file zero copy is not suppo…
giampaolo 7861737
use tempfiles
giampaolo f3eecfd
reset verbosity
giampaolo f67ce57
add test for smaller chunks
giampaolo d457254
add big file size test
giampaolo 8eb211d
add comment
giampaolo a0fe703
update doc
giampaolo 7296147
update whatsnew doc
giampaolo d0c3bba
update doc
giampaolo 2cafd80
catch Exception
giampaolo bb2a75f
remove unused import
giampaolo e5025dc
add test case for error on second sendfile() call
giampaolo a36a534
turn docstring into comment
giampaolo e9da3fa
add one more test
giampaolo 9fcc2e7
update comment
giampaolo 4f32242
add Misc/NEWS entry
giampaolo 24ad25a
get rid of COPY_BUFSIZE; it belongs to another PR
giampaolo 24d20e6
update doc
giampaolo 7b6e576
expose posix._fcopyfile() for OSX
giampaolo b82ddc9
Merge branch 'master' into shutil-osx-copyfile
giampaolo b62b61e
merge from linux branch
giampaolo 34e9618
merge from linux branch
giampaolo 6b20902
expose fcopyfile
giampaolo abf3ecb
arg clinic for the win implementation
giampaolo 91e492c
convert path type to path_t
giampaolo e02c69d
expose CopyFileW
giampaolo 73837e2
fix windows tests
giampaolo 28be4c1
release GIL
giampaolo 6c59adf
minor refactoring
giampaolo 700629d
update doc
giampaolo 077912e
update comment
giampaolo 62c6568
update docstrings
giampaolo a40a755
rename functions
giampaolo 7ba0085
rename test classes
giampaolo 6c96d97
update doc
giampaolo 80fbe6e
update doc
giampaolo fdf4bcb
update docstrings and comments
giampaolo 185f130
avoid do import nt|posix modules if unnecessary
giampaolo c8c98ae
set nt|posix modules to None if not available
giampaolo 17bb5e6
micro speedup
giampaolo d8b9bf9
update description
giampaolo b59ac57
add doc note
giampaolo 8eefce7
use better wording in doc
giampaolo 4fc8c6b
Merge branch 'master' into shutil-zero-copy
giampaolo 3048e3d
rename function using 'fastcopy' prefix instead of 'zerocopy'
giampaolo 11102e1
use :ref: in rst doc
giampaolo 7545273
change wording in doc
giampaolo 3261b74
add test to make sure sendfile() doesn't get called aymore in case it…
giampaolo 51c476d
move CopyFileW in _winapi and actually expose CopyFileExW instead
giampaolo 729dd23
fix line endings
giampaolo 1823828
add tests for mode bits
giampaolo a9d6a07
add docstring
giampaolo e3ce917
remove test file mode class; let's keep it for later when Istart addr…
giampaolo f81a0ec
update doc to reflect new changes
giampaolo 3e7475b
update doc
giampaolo 05dd3cf
adjust tests on win
giampaolo 9b54930
fix argument clinic error
giampaolo 2bec11c
update doc
giampaolo c87648f
OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags ar…
giampaolo 941f740
osx / copyfile: use path_t instead of char
giampaolo 4d28c12
do not set dst name in the OSError exception in order to remain consi…
giampaolo 2149b8b
add same file test
giampaolo 6a02a2a
add test for same file
giampaolo 2287508
have osx copyfile() pre-emptively check if src and dst are the same, …
giampaolo b9da5d5
turn PermissionError into appropriate SameFileError
giampaolo c921f46
expose ERROR_SHARING_VIOLATION in order to raise more appropriate Sam…
giampaolo bb24490
honour follow_symlinks arg when using CopyFileEx
giampaolo fef8b32
update Misc/NEWS
giampaolo 71be453
expose CreateDirectoryEx mock
giampaolo 6035fe2
change C type
giampaolo 8dc651e
CreateDirectoryExW actual implementation
giampaolo 5d0eada
provide specific makedirs() implementation for win
giampaolo d67cdc5
Merge branch 'shutil-zero-copy-8' of https://github.com/giampaolo/cpy…
giampaolo f65c8ae
fix typo
giampaolo 9c4508e
skeleton for SetNamedSecurityInfo
giampaolo bb1fee6
get security info for src path
giampaolo 566898a
finally set security attrs
giampaolo f435053
add unit tests
giampaolo 30c9a57
mimick os.makedirs() behavior and raise if dst dir exists
giampaolo 33f362f
set 2 paths for OSError object
giampaolo e17e729
set 2 paths for OSError object
giampaolo bc46f75
expand windows test
giampaolo cabbc02
in case of exception on os.sendfile() set filename and filename2 exce…
giampaolo d22ee08
set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX
giampaolo 7a08203
update doc
giampaolo ab284e9
do not use CreateDirectoryEx() in copytree() if source dir is a symli…
giampaolo ac9479d
use bytearray() and readinto()
giampaolo fd77a7e
use memoryview() with bytearray()
giampaolo 42a597e
refactoring + introduce a new _fastcopy_binfileobj() fun
giampaolo 5008a8d
remove CopyFileEx and other C wrappers
giampaolo e89dd20
remove code related to CopyFileEx
giampaolo c0dc4b8
Recognize binary files in copyfileobj()
giampaolo 29b9730
set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable
giampaolo a1bed32
use ctx manager for memoryview()
giampaolo d9d27a7
update doc
giampaolo 17bd78b
remove outdated doc
giampaolo b1d4917
remove last CopyFileEx remnants
giampaolo 5ce94e4
OSX - use fcopyfile(3) instead of copyfile(3)
giampaolo 07bcef5
update doc
giampaolo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
minor refactoring
- Loading branch information
commit 114c4dec91206f74ad5b86a63f04cedf969a4b25
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given a configure check for
_HAS_SENDFILE, the initial check for both it and_HAS_FCOPYFILEcan be moved up to the module level, and define_copyfileobj_fastdifferently based on the results. If both flags are false, then_copyfileobj_fastcan be aliased directly tocopyfileobj. (I believe Mac OS X defines both operations, so it will be important to give_HAS_FCOPYFILEprecedence when deciding which implementation of_copyfileobj_fastto use)The dynamic check to handle the case where a Python built on a newer distro is run against an older kernel (and hence still gets
ENOTSOCKat runtime) can then also be handled by setting_copyfileobj_fast = copyfileobjrather than by setting a flag that needs to be checked on every call.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See my comment above.