Skip to content

Commit e24f96a

Browse files
committed
Issue10050 - urlretrieve uses newer urlopen. reporthook of urlretrieve takes, block number, block read size, file_size
1 parent a2251aa commit e24f96a

5 files changed

Lines changed: 119 additions & 42 deletions

File tree

‎Doc/howto/urllib2.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,13 @@ The simplest way to use urllib.request is as follows::
5656
response = urllib.request.urlopen('http://python.org/')
5757
html = response.read()
5858

59+
If you wish to retrieve a resource via URL and store it in a temporary location,
60+
you can do so via the :func:`urlretrieve` function::
61+
62+
import urllib.request
63+
local_filename, headers = urllib.request.urlretrieve('http://python.org/')
64+
html = open(local_filename)
65+
5966
Many uses of urllib will be that simple (note that instead of an 'http:' URL we
6067
could have used an URL starting with 'ftp:', 'file:', etc.). However, it's the
6168
purpose of this tutorial to explain the more complicated cases, concentrating on

‎Doc/library/urllib.request.rst‎

Lines changed: 22 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1124,16 +1124,14 @@ The following functions and classes are ported from the Python 2 module
11241124
``urllib`` (as opposed to ``urllib2``). They might become deprecated at
11251125
some point in the future.
11261126

1127-
11281127
.. function:: urlretrieve(url, filename=None, reporthook=None, data=None)
11291128

1130-
Copy a network object denoted by a URL to a local file, if necessary. If the URL
1131-
points to a local file, or a valid cached copy of the object exists, the object
1132-
is not copied. Return a tuple ``(filename, headers)`` where *filename* is the
1129+
Copy a network object denoted by a URL to a local file. If the URL
1130+
points to a local file, the object will not be copied unless filename is supplied.
1131+
Return a tuple ``(filename, headers)`` where *filename* is the
11331132
local file name under which the object can be found, and *headers* is whatever
11341133
the :meth:`info` method of the object returned by :func:`urlopen` returned (for
1135-
a remote object, possibly cached). Exceptions are the same as for
1136-
:func:`urlopen`.
1134+
a remote object). Exceptions are the same as for :func:`urlopen`.
11371135

11381136
The second argument, if present, specifies the file location to copy to (if
11391137
absent, the location will be a tempfile with a generated name). The third
@@ -1144,32 +1142,39 @@ some point in the future.
11441142
third argument may be ``-1`` on older FTP servers which do not return a file
11451143
size in response to a retrieval request.
11461144

1145+
The following example illustrates the most common usage scenario::
1146+
1147+
>>> import urllib.request
1148+
>>> local_filename, headers = urllib.request.urlretrieve('http://python.org/')
1149+
>>> html = open(local_filename)
1150+
>>> html.close()
1151+
11471152
If the *url* uses the :file:`http:` scheme identifier, the optional *data*
1148-
argument may be given to specify a ``POST`` request (normally the request type
1149-
is ``GET``). The *data* argument must in standard
1150-
:mimetype:`application/x-www-form-urlencoded` format; see the :func:`urlencode`
1151-
function below.
1153+
argument may be given to specify a ``POST`` request (normally the request
1154+
type is ``GET``). The *data* argument must in standard
1155+
:mimetype:`application/x-www-form-urlencoded` format; see the
1156+
:func:`urlencode` function below.
11521157

11531158
:func:`urlretrieve` will raise :exc:`ContentTooShortError` when it detects that
11541159
the amount of data available was less than the expected amount (which is the
11551160
size reported by a *Content-Length* header). This can occur, for example, when
11561161
the download is interrupted.
11571162

11581163
The *Content-Length* is treated as a lower bound: if there's more data to read,
1159-
:func:`urlretrieve` reads more data, but if less data is available, it raises
1160-
the exception.
1164+
urlretrieve reads more data, but if less data is available, it raises the
1165+
exception.
11611166

11621167
You can still retrieve the downloaded data in this case, it is stored in the
11631168
:attr:`content` attribute of the exception instance.
11641169

1165-
If no *Content-Length* header was supplied, :func:`urlretrieve` can not check
1166-
the size of the data it has downloaded, and just returns it. In this case
1167-
you just have to assume that the download was successful.
1170+
If no *Content-Length* header was supplied, urlretrieve can not check the size
1171+
of the data it has downloaded, and just returns it. In this case you just have
1172+
to assume that the download was successful.
11681173

11691174
.. function:: urlcleanup()
11701175

1171-
Clear the cache that may have been built up by previous calls to
1172-
:func:`urlretrieve`.
1176+
Cleans up temporary files that may have been left behind by previous
1177+
calls to :func:`urlretrieve`.
11731178

11741179
.. class:: URLopener(proxies=None, **x509)
11751180

‎Lib/test/test_urllib.py‎

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -384,11 +384,11 @@ def test_copy(self):
384384

385385
def test_reporthook(self):
386386
# Make sure that the reporthook works.
387-
def hooktester(count, block_size, total_size, count_holder=[0]):
388-
self.assertIsInstance(count, int)
389-
self.assertIsInstance(block_size, int)
390-
self.assertIsInstance(total_size, int)
391-
self.assertEqual(count, count_holder[0])
387+
def hooktester(block_count, block_read_size, file_size, count_holder=[0]):
388+
self.assertIsInstance(block_count, int)
389+
self.assertIsInstance(block_read_size, int)
390+
self.assertIsInstance(file_size, int)
391+
self.assertEqual(block_count, count_holder[0])
392392
count_holder[0] = count_holder[0] + 1
393393
second_temp = "%s.2" % support.TESTFN
394394
self.registerFileForCleanUp(second_temp)
@@ -399,8 +399,8 @@ def hooktester(count, block_size, total_size, count_holder=[0]):
399399
def test_reporthook_0_bytes(self):
400400
# Test on zero length file. Should call reporthook only 1 time.
401401
report = []
402-
def hooktester(count, block_size, total_size, _report=report):
403-
_report.append((count, block_size, total_size))
402+
def hooktester(block_count, block_read_size, file_size, _report=report):
403+
_report.append((block_count, block_read_size, file_size))
404404
srcFileName = self.createNewTempFile()
405405
urllib.request.urlretrieve(self.constructLocalFileUrl(srcFileName),
406406
support.TESTFN, hooktester)
@@ -410,31 +410,31 @@ def hooktester(count, block_size, total_size, _report=report):
410410
def test_reporthook_5_bytes(self):
411411
# Test on 5 byte file. Should call reporthook only 2 times (once when
412412
# the "network connection" is established and once when the block is
413-
# read). Since the block size is 8192 bytes, only one block read is
414-
# required to read the entire file.
413+
# read).
415414
report = []
416-
def hooktester(count, block_size, total_size, _report=report):
417-
_report.append((count, block_size, total_size))
415+
def hooktester(block_count, block_read_size, file_size, _report=report):
416+
_report.append((block_count, block_read_size, file_size))
418417
srcFileName = self.createNewTempFile(b"x" * 5)
419418
urllib.request.urlretrieve(self.constructLocalFileUrl(srcFileName),
420419
support.TESTFN, hooktester)
421420
self.assertEqual(len(report), 2)
422-
self.assertEqual(report[0][1], 8192)
423-
self.assertEqual(report[0][2], 5)
421+
self.assertEqual(report[0][1], 0)
422+
self.assertEqual(report[1][1], 5)
424423

425424
def test_reporthook_8193_bytes(self):
426425
# Test on 8193 byte file. Should call reporthook only 3 times (once
427426
# when the "network connection" is established, once for the next 8192
428427
# bytes, and once for the last byte).
429428
report = []
430-
def hooktester(count, block_size, total_size, _report=report):
431-
_report.append((count, block_size, total_size))
429+
def hooktester(block_count, block_read_size, file_size, _report=report):
430+
_report.append((block_count, block_read_size, file_size))
432431
srcFileName = self.createNewTempFile(b"x" * 8193)
433432
urllib.request.urlretrieve(self.constructLocalFileUrl(srcFileName),
434433
support.TESTFN, hooktester)
435434
self.assertEqual(len(report), 3)
436-
self.assertEqual(report[0][1], 8192)
437-
self.assertEqual(report[0][2], 8193)
435+
self.assertEqual(report[0][1], 0)
436+
self.assertEqual(report[1][1], 8192)
437+
self.assertEqual(report[2][1], 1)
438438

439439

440440
class urlretrieve_HttpTests(unittest.TestCase, FakeHTTPMixin):

‎Lib/urllib/request.py‎

Lines changed: 72 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,9 @@
9494
import sys
9595
import time
9696
import collections
97+
import tempfile
98+
import contextlib
99+
97100

98101
from urllib.error import URLError, HTTPError, ContentTooShortError
99102
from urllib.parse import (
@@ -156,17 +159,78 @@ def install_opener(opener):
156159
global _opener
157160
_opener = opener
158161

159-
# TODO(jhylton): Make this work with the same global opener.
160-
_urlopener = None
162+
_url_tempfiles = []
161163
def urlretrieve(url, filename=None, reporthook=None, data=None):
162-
global _urlopener
163-
if not _urlopener:
164-
_urlopener = FancyURLopener()
165-
return _urlopener.retrieve(url, filename, reporthook, data)
164+
"""
165+
Retrieve a URL into a temporary location on disk.
166+
167+
Requires a URL argument. If a filename is passed, it is used as
168+
the temporary file location. The reporthook argument should be
169+
a callable that accepts a block number, a read size, and the
170+
total file size of the URL target. The data argument should be
171+
valid URL encoded data.
172+
173+
If a filename is passed and the URL points to a local resource,
174+
the result is a copy from local file to new file.
175+
176+
Returns a tuple containing the path to the newly created
177+
data file as well as the resulting HTTPMessage object.
178+
"""
179+
url_type, path = splittype(url)
180+
181+
with contextlib.closing(urlopen(url, data)) as fp:
182+
headers = fp.info()
183+
184+
# Just return the local path and the "headers" for file://
185+
# URLs. No sense in performing a copy unless requested.
186+
if url_type == "file" and not filename:
187+
return os.path.normpath(path), headers
188+
189+
# Handle temporary file setup.
190+
if filename:
191+
tfp = open(filename, 'wb')
192+
else:
193+
tfp = tempfile.NamedTemporaryFile(delete=False)
194+
filename = tfp.name
195+
_url_tempfiles.append(filename)
196+
197+
with tfp:
198+
result = filename, headers
199+
bs = 1024*8
200+
size = -1
201+
read = 0
202+
blocknum = 0
203+
if "content-length" in headers:
204+
size = int(headers["Content-Length"])
205+
206+
if reporthook:
207+
reporthook(blocknum, 0, size)
208+
209+
while True:
210+
block = fp.read(bs)
211+
if not block:
212+
break
213+
read += len(block)
214+
tfp.write(block)
215+
blocknum += 1
216+
if reporthook:
217+
reporthook(blocknum, len(block), size)
218+
219+
if size >= 0 and read < size:
220+
raise ContentTooShortError(
221+
"retrieval incomplete: got only %i out of %i bytes"
222+
% (read, size), result)
223+
224+
return result
166225

167226
def urlcleanup():
168-
if _urlopener:
169-
_urlopener.cleanup()
227+
for temp_file in _url_tempfiles:
228+
try:
229+
os.unlink(temp_file)
230+
except EnvironmentError:
231+
pass
232+
233+
del _url_tempfiles[:]
170234
global _opener
171235
if _opener:
172236
_opener = None

‎Misc/ACKS‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -288,6 +288,7 @@ Julien Élie
288288
Lance Ellinghaus
289289
David Ely
290290
Jeff Epler
291+
Jeff McNeil
291292
Tom Epperly
292293
Stoffel Erasmus
293294
Jürgen A. Erhard

0 commit comments

Comments
 (0)