transparent gzip compression in urllib

@rhettinger

BPO	1508475
Nosy	@rhettinger, @jcea, @orsenthil, @vstinner, @merwok, @berkerpeksag, @vadmium, @JimJJewett, @serhiy-storchaka, @demianbrecht
Files	urllib2-gzip.patch: urllib2-gzip.patch issue1508475.diff http_client_gzip.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = None
created_at = <Date 2006-06-19.08:59:09.000>
labels = ['type-feature', 'library']
title = 'transparent gzip compression in urllib'
updated_at = <Date 2015-03-07.02:32:39.334>
user = 'https://bugs.python.org/antialize'

bugs.python.org fields:

activity = <Date 2015-03-07.02:32:39.334>
actor = 'demian.brecht'
assignee = 'orsenthil'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2006-06-19.08:59:09.000>
creator = 'antialize'
dependencies = []
files = ['7335', '19811', '34177']
hgrepos = []
issue_num = 1508475
keywords = ['patch']
message_count = 23.0
messages = ['50500', '50501', '114671', '114725', '114726', '122342', '122343', '122351', '158315', '158380', '158400', '160355', '160384', '163925', '163935', '211893', '212252', '212473', '213982', '226258', '226399', '234363', '236013']
nosy_count = 18.0
nosy_names = ['rhettinger', 'jjlee', 'jcea', 'orsenthil', 'jerub', 'vstinner', 'antialize', 'ruseel', 'nadeem.vawda', 'thomaspinckney3', 'eric.araujo', 'abacabadabacaba', 'jcon', 'berker.peksag', 'martin.panter', 'Jim.Jewett', 'serhiy.storchaka', 'demian.brecht']
pr_nums = []
priority = 'high'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1508475'
versions = ['Python 3.5']

Linked PRs

gh-43521: Add an opt-in gzip Content-Encoding handler to urllib.request #154197

Some webservers support gzipping things before sending
them, this patch adds transparrent support for this in
urllib2 (documentation http://www.http-compression.com/)

This patach *requires* hash patch 914340 as a
prerequirement as this enabels stream support in the
gzip libary..

Looks good.

This needs tests and docs. As a new feature, this could not be released until Python 2.6.

It would be nice to have support for managing content negotiation in general, but that wish isn't an obstacle to this patch.

@jakob

@jakob could you provide an updated patch for py3k that includes unit test and doc changes?

No, I have long since moved on to other things.

Its okay, Jacab, we will take it forward.

I think, the transparent compression should work at http.client level. I also agree with other points made by Serhiy:

transparent decompression should delete headers Content-Encoding and Content-Length (this is as per RFC too)
Should not do another compression if the user has a explicit specified intent of using Content-Encoding: gzip and is ready to do decompression himself.
This transparent compression/decompression would require the availability gzip module, if not then the feature may be disabled and normal request-response cycle would proceed.
I think, having it 'ON' with a flag to switch 'OFF' would be more desirable than having this feature via Handler. The reason being it can help in performance of any requests on servers that support it and browsers have adopted similar approach too.

I updated bpo-1508475.diff for Python 3.4 and removed the change in urllib: http_client_gzip.patch. The patch only changes http.client to support server sending gzip data.

For example, the new python.org website serves gzip data even if the Accept-Encoding header is not sent by the client: see the issue bpo-20719.

I have code that already handles an “gzip” encoded response from urlopen(). All three patches leave the Content-Encoding header intact, so I suspect my code would try to decompress the body a second time. Deleting this header (as already suggested) would work for me.

Victor, the patch looks good and would be a welcome enhancement.

There should be an option for turning this on and off (perhaps, I want the zipped content and want to unzip later or in a different thread).

Consider adding support for "deflate" as well.

This is an enhancement, so I am changing the affected version from 3.3 to 3.5.

It is python-only, which works well with the cheeseshop.

That said, the patch is truly short; if that is really sufficient, it could almost go into the documentation as a recipe. But I would prefer some more assurances that it actually does work; a quick skim suggests that it relies on a superclass happening to implement read via readinto.

Needs tests and documentation change.

I think the patch is indeed a bit short, for instannce it looks like calling read() without a size limit could bypass the decoding.

Also, I wonder if Content-Encoding handling is better done at a higher level. What if someone wants to download a *.tar.gz file? They may not expect the tar file to be transparently decompressed. And I suspect this would blow up if you tried a partial range request.

Transfer-Encoding is meant to be the proper way to transparently compress HTTP messages at a low level, but it doesn’t seem to be used as much in the real world.

Related: bpo-1243678, which includes a patch for “httplib” (now known as “http.client”?). That patch looks like it sets Accept-Encoding and decodes according to Content-Encoding. However I suspect it is also trying to be too “transparent” at the wrong level and would have many of the problems already mentioned here.

The Lib/xmlrpc/client.py file appears to already support compression using “Content-Encoding: gzip”. Perhaps it could be leveraged for any work on this issue.

I suggest resolving bpo-15955 first, then the GzipFile API could be used without fear of decompression bombs.

Serhiy, when this came up for bulk-closing you said transparent gzip was a nice-to-have "if anybody provide PRs" — though not easy. Fair enough on that — it never really stalled on code, it stalled on one design call: opt-in handler vs default-on at http.client. That call's yours; here's one concrete answer to it: #154197.

I went opt-in — a HTTPGzipHandler that's not in build_opener()'s defaults, so urlopen() is unchanged unless you ask for it. That sidesteps Martin's 2014 footgun, where default-on would silently decompress a .tar.gz and break Range requests. It decodes only when it added the Accept-Encoding itself, and strips Content-Encoding/Content-Length.

Senthil preferred default-on with a flag; I can flip it if you'd rather — the mechanism's identical. gzip only for now, deflate a follow-up.

Merge remote-tracking branch 'upstream/main' into pythongh-43521-gzip…

Navigation Menu

Repository navigation

transparent gzip compression in urllib #43521

Linked PRs

15 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

transparent gzip compression in urllib #43521

Description

Linked PRs

Activity

antialize commented on Jun 19, 2006

jjlee commented on Jan 30, 2007

BreamoreBoy commented on Aug 22, 2010

antialize commented on Aug 23, 2010

orsenthil commented on Aug 23, 2010

15 remaining items

orsenthil commented on Jun 25, 2012

vstinner commented on Feb 21, 2014

vadmium commented on Feb 26, 2014

rhettinger commented on Feb 28, 2014

jimjjewett commented on Mar 18, 2014

vadmium commented on Sep 2, 2014

vadmium commented on Sep 5, 2014

vadmium commented on Jan 20, 2015

vadmium commented on Feb 15, 2015

soreavis commented on Jul 19, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions