Repository navigation
Support iterable bodies in httplib #47493
Description
| BPO | 3243 |
|---|---|
| Nosy | @birkenfeld, @rhettinger, @orsenthil, @pitrou, @catlee, @merwok, @IIIIllllIIIIllllIIIIllllIIIIllllIIIIll, @sorcio, @tzickel |
| PRs | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/orsenthil'
closed_at = <Date 2011-01-15.18:26:50.823>
created_at = <Date 2008-06-30.18:01:18.309>
labels = ['easy', 'type-feature', 'library']
title = 'Support iterable bodies in httplib'
updated_at = <Date 2018-10-30.20:47:43.149>
user = 'https://github.com/catlee'bugs.python.org fields:
activity = <Date 2018-10-30.20:47:43.149>
actor = 'tzickel'
assignee = 'orsenthil'
closed = True
closed_date = <Date 2011-01-15.18:26:50.823>
closer = 'orsenthil'
components = ['Library (Lib)']
creation = <Date 2008-06-30.18:01:18.309>
creator = 'catlee'
dependencies = []
files = ['12241', '19852', '19873', '19879', '19890', '19891', '19900', '19901', '19902']
hgrepos = []
issue_num = 3243
keywords = ['patch', 'easy', 'needs review']
message_count = 62.0
messages = ['69014', '77037', '84304', '100817', '110675', '120999', '122647', '122707', '122756', '122784', '122787', '122819', '122822', '122878', '122898', '122904', '122905', '122906', '122907', '122909', '122910', '122911', '122912', '122915', '122921', '122970', '122973', '122986', '122987', '122996', '122998', '122999', '123001', '123038', '123039', '123040', '123043', '123044', '123045', '123046', '123047', '123049', '123050', '123051', '123052', '123053', '123069', '123073', '124339', '124340', '124341', '124342', '124343', '124344', '124345', '124355', '124588', '126326', '126342', '126344', '126346', '328946']
nosy_count = 11.0
nosy_names = ['jhylton', 'georg.brandl', 'rhettinger', 'orsenthil', 'pitrou', 'catlee', 'eric.araujo', 'rcoyner', 'xuanji', 'davide.rizzo', 'tzickel']
pr_nums = ['10226']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue3243'
versions = ['Python 3.2']
Activity
catlee commentedon Jun 30, 2008
httplib should support requests whose bodies are iterable objects. This
would facilitate doing large file uploads via HTTP since you wouldn't
have to load the entire file into memory to create the request string.
Index: Lib/httplib.py
===================================================================
--- Lib/httplib.py (revision 64600)
+++ Lib/httplib.py (working copy)
@@ -688,7 +688,12 @@
self.__state = _CS_IDLE
catlee commentedon Dec 5, 2008
The attached patch implements this for python 2.7. It also adds support
for iterable bodies in urllib2, where it is more generally useful.
urllib2 enforces the presence of a Content-Length header in the request
if the body is an iterable, whereas httplib does not.
The patch also includes updates to docs and tests (which all pass as of
r67584 on my macbook)
jhylton commentedon Mar 28, 2009
Seems like a reasonable feature request. I'm going to apply a variant
of the patch in 3.1 first.
rcoyner commentedon Mar 11, 2010
This patch and its tests still work. Any particular reason why it hasn't been adopted yet?
BreamoreBoy commentedon Jul 18, 2010
Could someone with knowledge of httplib please move this forward, all the comments I see are positive.
merwok commentedon Nov 12, 2010
The patch needs to be ported to the py3k branch.
IIIIllllIIIIllllIIIIllllIIIIllllIIIIll commentedon Nov 28, 2010
Hi, attached is a ported version of the patch that applies against py3k.
rhettinger commentedon Nov 28, 2010
Instead of
hasattr(str,'next')
consider using
isinstance(str, collections.Iterable)
Also consider changing the variable name from the now overly type specific, "str" to something like "source" to indicate the significance of the data, not its type.
IIIIllllIIIIllllIIIIllllIIIIllllIIIIll commentedon Nov 29, 2010
Hi Raymond, I assume you're referring to catlee's patch? 'str' has been changed to 'data' in python 3.2
While porting the patch I ran into this issue, which is that isinstance(str, collections.Iterable) doesn't behave exactly like hasattr(str,'next')
so using the isinstance method would actually match against strings, which iirc would make the program fail. I can confirm later if needed.
44 remaining items
IIIIllllIIIIllllIIIIllllIIIIllllIIIIll commentedon Dec 19, 2010
Also, I am not familiar with the backward-comparability requirements of py3k, but orsenthil's patch will break py3k code that relies on data being a string, shouldn't this be mentioned somewhere?
Finally, I do not understand why my proposed change, which is to add
+ if not data:
+ request.add_unredirected_header('Content-length', '0')
so that code that relies on being able to pass a blank string as data (for instance, the code in test_urllib2.py) is still able to do so.
birkenfeld commentedon Dec 19, 2010
Hmm, indeed: Senthil, could data be a string in earlier versions?
If yes, the code should be changed to still allow that. (But after beta2 please, it's already tagged.)
birkenfeld commentedon Dec 19, 2010
Raising priority so that this gets sorted out before final.
orsenthil commentedon Dec 19, 2010
Xuanji, Thanks for the comments on 'data' being bytes. I had just cared to add the feature information. I think that data detail should have been updated too.
I think for your other two questions, we discussed it msg123051 - socket in py3k handles only bytes, sending string was wrong and test_urllib2 had the mistake in sending zero length strings which weren't detected.
However, let me see if we have to accommodate those "very special case" where data can be a zero length string just to accommodate the mistake it was present in the earlier version.
orsenthil commentedon Dec 24, 2010
A mistake with Content-Length in the previous commit resolved in revision 87469.
birkenfeld commentedon Jan 15, 2011
Anything left to do here, Senthil?
orsenthil commentedon Jan 15, 2011
Hi Georg,
In the previous comment, I had written that
'let me see if we have to accommodate those "very special case"
where data can be a zero length string just to accommodate the mistake
it was present in the earlier version.'
More, I think about it, the more it seems to me that accommodating that
special wrong case is not required.
The decision boils down to this.
In py3k, data for POST should be bytes.
But urllib.request had a 'bug/hole' that when a zero length string
was passed as a data, it did not raise an exception. There were cases
in test_urllib2 where zero length string was passed. I argue that it
was more of a mistake than, what we actually wanted to test it.
Because there is NO practical scenario where Zero length data as a
POST is useful.
this zero length string would raise an Exception and would demand that
even if it is zero length, please send it as bytes.
IMO, this is correct behavior and we need not accommodate the previous
one.
So, I would recommend closing this bug as Fixed without further change.
At most, a NEWS item can be added to explain point 2.
IIIIllllIIIIllllIIIIllllIIIIllllIIIIll commentedon Jan 15, 2011
Yes, I think we should close it.
orsenthil commentedon Jan 15, 2011
Thanks for the note. I shall some details to the NEWS entry before the release. Closing this report.
tzickel commentedon Oct 30, 2018
This patch was opened for 2.7 but never applied there ?
#10226
This causes a bug with requests HTTP library (and others as well as httplib) when you want to send an iterable object as POST data (with a non-chunked way), it works in Python 3 but not 2, and this effects behaviour and performance...