Repository navigation
Expose the length arg from shutil.copyfileobj for public use #73845
Description
| BPO | 29659 |
|---|---|
| Nosy | @rhettinger, @terryjreedy, @methane, @serhiy-storchaka, @tgoodlet |
| PRs | length arg for public use #328 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-02-25.09:27:42.340>
created_at = <Date 2017-02-27.00:57:24.705>
labels = ['3.8', 'type-feature', 'library']
title = 'Expose the `length` arg from shutil.copyfileobj for public use'
updated_at = <Date 2019-02-25.09:27:42.339>
user = 'https://github.com/tgoodlet'bugs.python.org fields:
activity = <Date 2019-02-25.09:27:42.339>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2019-02-25.09:27:42.340>
closer = 'methane'
components = ['Library (Lib)']
creation = <Date 2017-02-27.00:57:24.705>
creator = 'goodboy'
dependencies = []
files = []
hgrepos = []
issue_num = 29659
keywords = []
message_count = 5.0
messages = ['288616', '288962', '288974', '305625', '336504']
nosy_count = 5.0
nosy_names = ['rhettinger', 'terry.reedy', 'methane', 'serhiy.storchaka', 'goodboy']
pr_nums = ['328']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29659'
versions = ['Python 3.8']
Activity
tgoodlet commentedon Feb 27, 2017
It would be handy to be able to pass the
lengthdefault argument defined byshutil.copyfileobj()to other public functions througout the module. When copying very large files (1GB +), increasing the memory buffer can divide the copy time in half and currently the only way to accomplish this is by overriding the needed function(s) which callshutil.copyfileobj().I propose a simple non-invasive change where the following functions will also expose the
lengthkwarg and pass it downwards tocopyfileobj:rhettinger commentedon Mar 4, 2017
This seems reasonable to me.
methane commentedon Mar 4, 2017
How about increasing default value to 32KiB or 64KiB too?
binutils's cp works well for most cases, while it doesn't have option to specify blocksize.
I want Python's copy functions works nice enough for common cases too.
binutils cp uses 128KiB block for normal file. see more information in https://github.com/coreutils/coreutils/blob/master/src/ioblksize.h
But copyfileobj can be used other file-likes.
On Linux, typical socket buffer size is 128KiB, and typical pipe buffer size is 64KiB. So block size larger than 64KiB may cause unneeded blocking.
That's why I suggested 32KiB or 64KiB.
serhiy-storchaka commentedon Nov 6, 2017
Could you please provide any benchmarks Tyler?
methane commentedon Feb 25, 2019
bpo-33671 fixed this.