Repository navigation
Use sendfile where possible in httplib #57768
Description
| BPO | 13559 |
|---|---|
| Nosy | @orsenthil, @giampaolo, @tiran, @benjaminp, @merwok, @vadmium, @moreati |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2011-12-08.20:47:25.389>
labels = ['library', '3.10', 'performance']
title = 'Use sendfile where possible in httplib'
updated_at = <Date 2021-03-09.10:40:33.692>
user = 'https://github.com/benjaminp'bugs.python.org fields:
activity = <Date 2021-03-09.10:40:33.692>
actor = 'christian.heimes'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2011-12-08.20:47:25.389>
creator = 'benjamin.peterson'
dependencies = ['17552', '23740']
files = ['35569']
hgrepos = []
issue_num = 13559
keywords = ['patch']
message_count = 13.0
messages = ['149052', '149073', '149075', '149077', '149078', '220262', '250435', '348634', '387699', '387729', '387751', '387754', '388346']
nosy_count = 9.0
nosy_names = ['orsenthil', 'giampaolo.rodola', 'christian.heimes', 'benjamin.peterson', 'eric.araujo', 'rosslagerwall', 'kasun', 'martin.panter', 'Alex.Willmer']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue13559'
versions = ['Python 3.10']
Activity
benjaminp commentedon Dec 8, 2011
HTTPConnection.send() should use os.sendfile when possible to avoid copying data into userspace and back.
giampaolo commentedon Dec 9, 2011
This is not possible for two reasons:
on most POSIX systems, sendfile() works with mmap-like ("regular") files only, while HTTPConnection.send() accepts any file-like object as long as it provides a read() method
after read()ing a chunk of data from the file and before send()ing it over the socket, the data can be subject to an intermediate conversion (datablock.encode("iso-8859-1")):
http://hg.python.org/cpython/file/87c6be1e393a/Lib/http/client.py#l839
...whereas sendfile() can only be used to send a binary file "as-is"
I think we can use sendfile() in ftplib.py though .
I'll open a ticket for that.
benjaminp commentedon Dec 9, 2011
2011/12/8 Giampaolo Rodola' <report@bugs.python.org>:
I presume you could check for a binary mode, though? Also, you can
catch EINVAl on invalid fds.
giampaolo commentedon Dec 9, 2011
ftplib's sendfile support is not tracked as bpo-13559.
Considerations I made there should apply here as well.
giampaolo commentedon Dec 9, 2011
Ops! I meant bpo-13564.
giampaolo commentedon Jun 11, 2014
Patch in attachment uses the newly added socket.sendfile() method (bpo-17552).
vadmium commentedon Sep 11, 2015
The multiple personalities of HTTPConnection.send() and friends is a bit of a can of worms. I suggest working on bpo-23740 to get an idea of what kinds of file objects are meant to be supported, and what things may work by accident and be used in the real world.
For instance, is it possible to manually set Content-Length, and say supply a GzipFile reader, or file object positioned halfway through the file? How does this interact with the socket.sendfile() call?
vstinner commentedon Jul 29, 2019
This issue is no newcomer friendly, I remove the "easy" keyword.
moreati commentedon Feb 26, 2021
I would like to take a stab at this. Giampaolo, would it be okay if I made a pull request updated from your patch? With the appropriate "Co-authored-by: Author Name <email_address>" line.
orsenthil commentedon Feb 26, 2021
Alex, https://bugs.python.org/issue23740 is identified as a dependency on this issue. We will have to resolve that first, and come back to this. And yes, if you contribute on other's patch, both the contributions will be included and appropriately credited.
moreati commentedon Feb 26, 2021
To check my understanding
Is the motivation for the closer to
My guess is 5.
orsenthil commentedon Feb 26, 2021
Yes, the point number 5. We will have to evaluate if sendfile side-steps and avoids the issues noted in bpo-23740
tiran commentedon Mar 9, 2021
sendfile() only works for plain HTTP. For technical reasons it does not work for HTTPS (*). These days majority of services use HTTPS. Therefore the usefulness of sendfile() patch is minimal.
(*) It is possible to use sendfile() for TLS connections, but the feature requires a Kernel module that provides kTLS offloading feature, https://www.kernel.org/doc/html/latest/networking/tls-offload.html . In user space it requires OpenSSL 3.0.0 with kTLS support. 3.0.0 is currently under development.