Expose the length arg from shutil.copyfileobj for public use

@rhettinger

BPO	29659
Nosy	@rhettinger, @terryjreedy, @methane, @serhiy-storchaka, @tgoodlet
PRs	bpo-29659: Expose copyfileobj() `length` arg for public use #328

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-02-25.09:27:42.340>
created_at = <Date 2017-02-27.00:57:24.705>
labels = ['3.8', 'type-feature', 'library']
title = 'Expose the `length` arg from shutil.copyfileobj for public use'
updated_at = <Date 2019-02-25.09:27:42.339>
user = 'https://github.com/tgoodlet'

bugs.python.org fields:

activity = <Date 2019-02-25.09:27:42.339>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2019-02-25.09:27:42.340>
closer = 'methane'
components = ['Library (Lib)']
creation = <Date 2017-02-27.00:57:24.705>
creator = 'goodboy'
dependencies = []
files = []
hgrepos = []
issue_num = 29659
keywords = []
message_count = 5.0
messages = ['288616', '288962', '288974', '305625', '336504']
nosy_count = 5.0
nosy_names = ['rhettinger', 'terry.reedy', 'methane', 'serhiy.storchaka', 'goodboy']
pr_nums = ['328']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29659'
versions = ['Python 3.8']

It would be handy to be able to pass the length default argument defined by shutil.copyfileobj() to other public functions througout the module. When copying very large files (1GB +), increasing the memory buffer can divide the copy time in half and currently the only way to accomplish this is by overriding the needed function(s) which call shutil.copyfileobj().

I propose a simple non-invasive change where the following functions will also expose the length kwarg and pass it downwards to copyfileobj:

copyfile
copy
copy2

This seems reasonable to me.

How about increasing default value to 32KiB or 64KiB too?

binutils's cp works well for most cases, while it doesn't have option to specify blocksize.
I want Python's copy functions works nice enough for common cases too.

binutils cp uses 128KiB block for normal file. see more information in https://github.com/coreutils/coreutils/blob/master/src/ioblksize.h

But copyfileobj can be used other file-likes.
On Linux, typical socket buffer size is 128KiB, and typical pipe buffer size is 64KiB. So block size larger than 64KiB may cause unneeded blocking.
That's why I suggested 32KiB or 64KiB.

Could you please provide any benchmarks Tyler?

bpo-33671 fixed this.

Navigation Menu

Repository navigation

Expose the `length` arg from shutil.copyfileobj for public use #73845

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Expose the length arg from shutil.copyfileobj for public use #73845

Description

Activity

tgoodlet commented on Feb 27, 2017

rhettinger commented on Mar 4, 2017

methane commented on Mar 4, 2017

serhiy-storchaka commented on Nov 6, 2017

methane commented on Feb 25, 2019

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Expose the `length` arg from shutil.copyfileobj for public use #73845