Skip to content

Add large-file tests for modules using sendfile(2) #81277

Closed
Closed
Add large-file tests for modules using sendfile(2)#81277
Labels
3.9 (EOL)end of lifetestsTests in the Lib/test dirtype-featureA feature request or enhancement

Description

@giampaolo
giampaolo
opened on May 30, 2019
BPO 37096
Nosy @vstinner, @giampaolo, @encukou, @stratakis
PRs
  • bpo-37096: Add large-file tests for modules using sendfile(2) #13676
    		•
    Files
  • disk-space.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-02-17.11:23:56.052>
created_at = <Date 2019-05-30.07:18:19.110>
labels = ['type-feature', 'tests', '3.9']
title = 'Add large-file tests for modules using sendfile(2)'
updated_at = <Date 2020-02-17.11:23:56.051>
user = 'https://github.com/giampaolo'

    bugs.python.org fields:

    activity = <Date 2020-02-17.11:23:56.051>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-02-17.11:23:56.052>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2019-05-30.07:18:19.110>
creator = 'giampaolo.rodola'
dependencies = []
files = ['48636']
hgrepos = []
issue_num = 37096
keywords = ['patch']
message_count = 14.0
messages = ['343951', '353533', '353552', '353699', '353701', '353703', '353704', '353709', '356368', '357859', '357863', '357944', '360325', '362136']
nosy_count = 5.0
nosy_names = ['vstinner', 'giampaolo.rodola', 'petr.viktorin', 'rosslagerwall', 'cstratak']
pr_nums = ['13676']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37096'
versions = ['Python 3.9']

    Activity

    1. giampaolo commented on May 30, 2019

      @giampaolo
      giampaolo
      on May 30, 2019
      ContributorAuthor

      The need for this emerged in:
      https://bugs.python.org/issue36610#msg343948.
      We currently use sendfile(2) syscall in high-level shutil.copyfile() on Linux and socket.sendfile() on UNIX. In addition this PR also tests shutil.copyfile() implementation on OSX which takes advantage of fcopyfile(2) syscall. The goal is to make sure "fast-copy" syscalls can handle files > 2G. On my Ubuntu 18.04 with SSD disk this introduces a 5 secs slowdown to the test run.

      According to:
      golang/go#13892
      ...problems may arise on SunOS, so if a BB will turn red this is expected.

    2. giampaolo
      added
      3.8 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      on May 30, 2019

    3. giampaolo commented on Sep 30, 2019

      @giampaolo
      giampaolo
      on Sep 30, 2019
      ContributorAuthor

      New changeset 5bcc6d8 by Giampaolo Rodola in branch 'master':
      bpo-37096: Add large-file tests for modules using sendfile(2) (GH-13676)
      5bcc6d8

    4. giampaolo
      added
      3.9 (EOL)end of life
      testsTests in the Lib/test dir
      and removed
      3.8 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      on Sep 30, 2019
    5. giampaolo
      closed this as completedon Sep 30, 2019
    6. giampaolo
      added
      type-featureA feature request or enhancement
      on Sep 30, 2019

    7. vstinner commented on Sep 30, 2019

      @vstinner
      vstinner
      on Sep 30, 2019
      Member

      TestCopyfile fails on 32-bit system: bpo-38319.

    8. vstinner
      reopened this on Sep 30, 2019

    9. stratakis commented on Oct 1, 2019

      @stratakis
      stratakismannequin
      on Oct 1, 2019
      Mannequin

      After this change I get some disk space issues on the Fedora rawhide buildbot for the clang installed build only (and strangely enough not for the other jobs). There are currently around 9GB of free space there:

      https://buildbot.python.org/all/#/builders/188/builds/1110

      ======================================================================
      ERROR: test_tofromfile (test.test_array.DoubleTest)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64.clang-installed/build/target/lib/python3.9/test/test_array.py",
line 374, in test_tofromfile
    f.close()
OSError: [Errno 28] No space left on device

    10. giampaolo commented on Oct 1, 2019

      @giampaolo
      giampaolo
      on Oct 1, 2019
      ContributorAuthor

      Sorry about that. I'm attaching a patch which skips those tests if there's not enough free disk space. If you confirm it works I'll make a PR.

    11. vstinner commented on Oct 1, 2019

      @vstinner
      vstinner
      on Oct 1, 2019
      Member

      Should we decorate the new files with @requires('largefile', '...')? Documentation:

      largefile - It is okay to run some test that may create huge
            files.  These tests can take a long time and may
            consume >2 GiB of disk space temporarily.

      It may be interesting to add an option to regrtest to specify how much space can be used by tests, an option similar to -M/--memlimit which is for memory.

      I don't recall if largefile is enabled by default on buildbots or not :-p

    12. stratakis commented on Oct 1, 2019

      @stratakis
      stratakismannequin
      on Oct 1, 2019
      Mannequin

      It seems that the -uall argument is passed to regrtest invocation for the buildbot run [0] which invokes the largefile tests (including all the resource intensive tests). However when configure is run you can see: checking whether to enable large file support... no

      [0] https://github.com/python/cpython/blob/master/Lib/test/libregrtest/cmdline.py#L81

    13. vstinner commented on Oct 1, 2019

      @vstinner
      vstinner
      on Oct 1, 2019
      Member

      checking whether to enable large file support... no

      That's only at the C level to decide how to pass a file position.

      It's unrelated to the regrtest "largefile" resource.

    14. stratakis commented on Nov 11, 2019

      @stratakis
      stratakismannequin
      on Nov 11, 2019
      Mannequin

      Alright added some more disk space at the buildbots, however it seems that it is not related to that. The current Fedora rawhide buildbot has 19GB of free space and the test is still failing.

      I tested on the Fedora stable buildbot for which I reduced intentionally the free disk space to 14GB and it passed: https://buildbot.python.org/all/#/builders/233/builds/398

      So something else might be at play here.

    15. encukou commented on Dec 5, 2019

      @encukou
      encukou
      on Dec 5, 2019
      Member

      It seems that the test actually requires >4GB disk space, because it's copying a 2GB file from one place to another.

      Also, it's using a temporary directory. On Fedora /tmp is memory-backed by default, it doesn't use regular disk space.

    16. encukou commented on Dec 5, 2019

      @encukou
      encukou
      on Dec 5, 2019
      Member

      The test passes with a 8GB /tmp, but not a 6GB one.

      That could suggest the test needs 3*2GB disk space for temporary files, plus something extra.

    17. stratakis commented on Dec 6, 2019

      @stratakis
      stratakismannequin
      on Dec 6, 2019
      Mannequin

      Can we reopen the issue?

      Clearly this change modifies the expectations of free disk space for the temp files created by the tests.

      Or at least clarify that those tests require more than 6gb of free disk space in /tmp for unix

    18. vstinner commented on Jan 20, 2020

      @vstinner
      vstinner
      on Jan 20, 2020
      Member

      Right now, "AMD64 Fedora Rawhide Clang 3.x" worker is broken: "C compiler cannot create executables"
      https://bugs.python.org/issue39398

    19. vstinner commented on Feb 17, 2020

      @vstinner
      vstinner
      on Feb 17, 2020
      Member

      I close the issue. The test was fixed in bpo-39488:

      commit b39fb8e
      Author: Giampaolo Rodola <g.rodola@gmail.com>
      Date: Wed Feb 5 18:20:52 2020 +0100

      bpo-39488: Skip test_largefile tests if not enough disk space (GH-18261)
    20. vstinner
      closed this as completedon Feb 17, 2020
    21. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.9 (EOL)end of lifetestsTests in the Lib/test dirtype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions