Skip to content

shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system #82500

Closed
Closed
shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system#82500
Labels
3.8 (EOL)end of life3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directory

Description

@vstinner
vstinner
opened on Sep 30, 2019
BPO 38319
Nosy @vstinner, @giampaolo, @ambv, @serhiy-storchaka, @pablogsal, @nanjekyejoannah
PRs
  • bpo-38319: Fix shutil._fastcopy_sendfile(): set sendfile() max block size #16491
  • [3.8] bpo-38319: Fix shutil._fastcopy_sendfile(): set sendfile() max block size (GH-16491) #16506
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2019-10-02.14:39:28.135>
created_at = <Date 2019-09-30.07:37:51.943>
labels = ['3.8', 'library', '3.9']
title = 'shutil.copyfile(): os.sendfile() fails with OverflowError on 32-bit system'
updated_at = <Date 2019-10-02.14:39:28.134>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2019-10-02.14:39:28.134>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-02.14:39:28.135>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2019-09-30.07:37:51.943>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 38319
keywords = ['patch']
message_count = 7.0
messages = ['353549', '353551', '353569', '353633', '353637', '353645', '353738']
nosy_count = 6.0
nosy_names = ['vstinner', 'giampaolo.rodola', 'lukasz.langa', 'serhiy.storchaka', 'pablogsal', 'nanjekyejoannah']
pr_nums = ['16491', '16506']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38319'
versions = ['Python 3.8', 'Python 3.9']

    Linked PRs

    Activity

    1. vstinner commented on Sep 30, 2019

      @vstinner
      vstinner
      on Sep 30, 2019
      MemberAuthor

      Error on a 32-bit buildbot worker where ssize_t maximum = 2,147,483,647 (2**31-1) bytes = ~2.0 GiB.

      test_largefile uses:

      # size of file to create (>2 GiB; 2 GiB == 2,147,483,648 bytes)
size = 2_500_000_000

      x86 Gentoo Installed with X 3.x:
      https://buildbot.python.org/all/#/builders/103/builds/3162

      ======================================================================
      ERROR: test_it (test.test_largefile.TestCopyfile)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/test/test_largefile.py", line 160, in test_it
    shutil.copyfile(TESTFN, TESTFN2)
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/shutil.py", line 266, in copyfile
    _fastcopy_sendfile(fsrc, fdst)
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/shutil.py", line 145, in _fastcopy_sendfile
    sent = os.sendfile(outfd, infd, offset, blocksize)
OverflowError: Python int too large to convert to C ssize_t

      On Linux (Fedora 30), man sendfile shows me the prototype:

         ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);

      Extract of Lib/shutil.py:

      # Hopefully the whole file will be copied in a single call.
# sendfile() is called in a loop 'till EOF is reached (0 return)
# so a bufsize smaller or bigger than the actual file size
# should not make any difference, also in case the file content
# changes while being copied.
try:
    blocksize = max(os.fstat(infd).st_size, 2 ** 23)  # min 8MB
except Exception:
    blocksize = 2 ** 27  # 128MB

          offset = 0
    while True:
        try:
            sent = os.sendfile(outfd, infd, offset, blocksize)
        except OSError as err:
            ...
        else:
            if sent == 0:
                break  # EOF
            offset += sent

      Extract of the Linux implementation of os.sendfile():

          int in, out;
    Py_ssize_t ret;
    off_t offset;
    ...
    Py_ssize_t count;
    PyObject *offobj;
    static char *keywords[] = {"out", "in",
                                "offset", "count", NULL};
    if (!PyArg_ParseTupleAndKeywords(args, kwdict, "iiOn:sendfile",
            keywords, &out, &in, &offobj, &count))
        return NULL;
    ...
    if (!Py_off_t_converter(offobj, &offset))
        return NULL;
      do {
    Py_BEGIN_ALLOW_THREADS
    ret = sendfile(out, in, &offset, count);
    Py_END_ALLOW_THREADS
} while (ret < 0 && errno == EINTR && !(async_err = PyErr_CheckSignals()));

      with:

      static int
Py_off_t_converter(PyObject *arg, void *addr)
{
#ifdef HAVE_LARGEFILE_SUPPORT
    *((Py_off_t *)addr) = PyLong_AsLongLong(arg);
#else
    *((Py_off_t *)addr) = PyLong_AsLong(arg);
#endif
    if (PyErr_Occurred())
        return 0;
    return 1;
}

      I understand that the error comes from the 4th sendfile() parameter: "count". The C code (of the Linux implementation) uses the "n" format for Py_ssize_t: Python/getargs.c calls PyLong_AsSsize_t().

      On a 64-bit system, it's less likely to reach Py_ssize_t maximum value (max = 2**63-1), but it's easy to reach on a 32-bit system (max = 2**31-1).

    2. vstinner
      added
      3.9 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      on Sep 30, 2019

    3. vstinner commented on Sep 30, 2019

      @vstinner
      vstinner
      on Sep 30, 2019
      MemberAuthor

      Oh, it's likely a regression caused by:

      commit 5bcc6d8
      Author: Giampaolo Rodola <g.rodola@gmail.com>
      Date: Mon Sep 30 12:51:55 2019 +0800

      bpo-37096: Add large-file tests for modules using sendfile(2) (GH-13676)

      https://buildbot.python.org/all/#/builders/103/builds/3162

      configure:

      checking for sendfile... yes
      checking whether to enable large file support... yes

      pythoninfo:

      platform.libc_ver: glibc 2.29
      platform.platform: Linux-4.19.72-gentoo-i686-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_5000+-with-glibc2.29

    4. vstinner commented on Sep 30, 2019

      @vstinner
      vstinner
      on Sep 30, 2019
      MemberAuthor

      Similar failure on ARMv7 Debian buster 3.x:
      https://buildbot.python.org/all/#/builders/176/builds/1372

      pythoninfo:
      sys.maxsize: 2147483647

    5. giampaolo commented on Oct 1, 2019

      @giampaolo
      giampaolo
      on Oct 1, 2019
      Contributor

      New changeset 94e1650 by Giampaolo Rodola in branch 'master':
      bpo-38319: Fix shutil._fastcopy_sendfile(): set sendfile() max block size (GH-16491)
      94e1650

    6. giampaolo commented on Oct 1, 2019

      @giampaolo
      giampaolo
      on Oct 1, 2019
      Contributor

      Looks like it worked:
      https://buildbot.python.org/all/#/builders/176/builds/1383

    7. giampaolo
      added
      3.8 (EOL)end of life
      on Oct 1, 2019

    8. ambv commented on Oct 1, 2019

      @ambv
      ambv
      on Oct 1, 2019
      Contributor

      New changeset 938c00c by Łukasz Langa (Miss Islington (bot)) in branch '3.8':
      bpo-38319: Fix shutil._fastcopy_sendfile(): set sendfile() max block size (GH-16491) (bpo-16506)
      938c00c

    9. vstinner commented on Oct 2, 2019

      @vstinner
      vstinner
      on Oct 2, 2019
      MemberAuthor

      32-bit buildbots are back to green. Thanks for the fix!

    10. vstinner
      closed this as completedon Oct 2, 2019
    11. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    12. bdraco
      added a commit that references this issue on Jul 22, 2023

      pythongh-82500: Fix asyncio sendfile overflow on 32bit

      231b64f
    13. gvanrossum
      added a commit that references this issue on Jul 23, 2023

      gh-82500: Fix asyncio sendfile overflow on 32bit (#107056)

      9eeb4b4
    14. jtcave
      added a commit that references this issue on Jul 23, 2023

      pythongh-82500: Fix asyncio sendfile overflow on 32bit (python#107056)

      72f0be3
    15. mementum
      added a commit that references this issue on Jul 23, 2023

      pythongh-82500: Fix asyncio sendfile overflow on 32bit (python#107056)

      24d12aa
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.8 (EOL)end of life3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directory

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions