Skip to content

Use copy_file_range() in shutil.copyfile() (server-side copy) #81340

Closed
Closed
Use copy_file_range() in shutil.copyfile() (server-side copy)#81340
Labels
3.12only security fixesperformancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directory

Description

@giampaolo
giampaolo
opened on Jun 5, 2019
BPO 37159
Nosy @facundobatista, @ncoghlan, @vstinner, @giampaolo, @encukou, @albertz, @vadmium, @desbma, @pablogsal
Files
  • patch.diff
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2019-06-05.05:24:51.971>
labels = ['library', '3.9', 'performance']
title = 'Use copy_file_range() in shutil.copyfile() (server-side copy)'
updated_at = <Date 2020-12-29.13:05:56.586>
user = 'https://github.com/giampaolo'

    bugs.python.org fields:

    activity = <Date 2020-12-29.13:05:56.586>
actor = 'Albert.Zeyer'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-06-05.05:24:51.971>
creator = 'giampaolo.rodola'
dependencies = []
files = ['48392']
hgrepos = []
issue_num = 37159
keywords = ['patch']
message_count = 6.0
messages = ['344671', '344679', '344680', '344691', '344693', '383996']
nosy_count = 11.0
nosy_names = ['facundobatista', 'ncoghlan', 'vstinner', 'giampaolo.rodola', 'StyXman', 'petr.viktorin', 'neologix', 'Albert.Zeyer', 'martin.panter', 'desbma', 'pablogsal']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue37159'
versions = ['Python 3.9']

    Linked PRs

    Activity

    1. giampaolo commented on Jun 5, 2019

      @giampaolo
      giampaolo
      on Jun 5, 2019
      ContributorAuthor

      This is a follow up of bpo-33639 (zero-copy via sendfile()) and bpo-26828 (os.copy_file_range()). On [Linux 4.5 / glib 2.27] shutil.copyfile() will use os.copy_file_range() instead of os.sendfile(). According to my benchmarks performances are the same but when dealing with NFS copy_file_range() is supposed to attempt doing a server-side copy, meaning there will be no exchange of data between client and server, making the copy operation an order of magnitude faster.

      Before proceeding unit-tests for big-file support should be added first (bpo-37096). We didn't hit the 3.8 deadline but I actually prefer to land this in 3.9 as I want to experiment with it a bit (copy_file_range() is quite new, bpo-26828 is still a WIP).

    2. giampaolo
      added
      3.9 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      performancePerformance or resource usage
      on Jun 5, 2019
    3. giampaolo
      changed the title [-]Have shutil.copyfile() use copy_file_range()[/-] [+]Use copy_file_range() in shutil.copyfile() (server-side copy)[/+] on Jun 5, 2019

    4. vstinner commented on Jun 5, 2019

      @vstinner
      vstinner
      on Jun 5, 2019
      Member

      Oh, I already created https://bugs.python.org/issue37157

      Can we move the discussion there?

    5. giampaolo commented on Jun 5, 2019

      @giampaolo
      giampaolo
      on Jun 5, 2019
      ContributorAuthor

      bpo-37157 is for reflink / CoW copy, this one is not.

    6. vstinner commented on Jun 5, 2019

      @vstinner
      vstinner
      on Jun 5, 2019
      Member

      bpo-37157 is for reflink / CoW copy, this one is not.

      Oh sorry, it seems like I misunderstood copy_file_range(). So it doesn't use/support CoW?

    7. giampaolo commented on Jun 5, 2019

      @giampaolo
      giampaolo
      on Jun 5, 2019
      ContributorAuthor

      Nope, it doesn't (see man page). We can simply use FICLONE (cp does the same).

    8. albertz commented on Dec 29, 2020

      @albertz
      albertzmannequin
      on Dec 29, 2020
      Mannequin

      According to the man page of copy_file_range (https://man7.org/linux/man-pages/man2/copy_file_range.2.html), copy_file_range also should support copy-on-write:

        copy_file_range() gives filesystems an opportunity to implement
  "copy acceleration" techniques, such as the use of reflinks
  (i.e., two or more inodes that share pointers to the same copy-
  on-write disk blocks) or server-side-copy (in the case of NFS).

      Is this wrong?

      However, while researching more about FICLONE vs copy_file_range, I found e.g. this: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24399

      Which suggests that there are other problems with copy_file_range?

    9. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    10. illia-v commented on Apr 16, 2022

      @illia-v
      illia-v
      on Apr 16, 2022
      Contributor

      FYI, GNU Coreutils 9.0 (released in September 2021) changed cp to:

      • use copy offload via copy_file_range where available;
      • enable CoW by default.

      https://lists.gnu.org/archive/html/info-gnu/2021-09/msg00010.html

    11. iritkatriel
      added
      3.12only security fixes
      and removed
      3.9 (EOL)end of life
      on Aug 26, 2022
    12. barneygale
      added a commit that references this issue on Jun 14, 2024

      GH-73991: Add `pathlib.Path.copy()` (#119058)

      7c38097
    13. mrahtz
      added a commit that references this issue on Jun 30, 2024

      pythonGH-73991: Add `pathlib.Path.copy()` (python#119058)

      7e37fbe
    14. noahbkim
      added a commit that references this issue on Jul 11, 2024

      pythonGH-73991: Add `pathlib.Path.copy()` (python#119058)

      81d2303
    15. estyxx
      added a commit that references this issue on Jul 17, 2024

      pythonGH-73991: Add `pathlib.Path.copy()` (python#119058)

      595b213
    16. zooba
      added a commit that references this issue on Feb 3, 2025

      gh-81340: Use copy_file_range in shutil.copyfile copy functions (GH-9…

      a33dcb9

    17. zooba commented on Feb 3, 2025

      @zooba
      zooba
      on Feb 3, 2025
      Member

      I've merged the changes so they get a bit of a run in 3.14 alpha. Nobody (apart from the contributor) seemed willing to step up and say that it's definitely the right thing to do.

      If there are issues, feel free to revert or modify. This isn't "my" code, I don't need to be consulted.

    18. zooba
      closed this as completedon Feb 3, 2025
    19. srinivasreddy
      added a commit that references this issue on Feb 7, 2025

      pythongh-81340: Use copy_file_range in shutil.copyfile copy functions (…

      e43fa23
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.12only security fixesperformancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directory

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions