Repository navigation
timeit called from within Python should allow autoranging #50671
Description
| BPO | 6422 |
|---|---|
| Nosy | @rhettinger, @amauryfa, @ncoghlan, @pitrou, @vstinner, @rbtcollins, @merwok, @stevendaprano, @asvetlov, @csabella |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/stevendaprano'
closed_at = <Date 2019-03-28.09:50:03.233>
created_at = <Date 2009-07-05.15:15:10.511>
labels = ['type-feature', 'library']
title = 'timeit called from within Python should allow autoranging'
updated_at = <Date 2019-03-28.09:50:03.232>
user = 'https://bugs.python.org/scottdaniels'bugs.python.org fields:
activity = <Date 2019-03-28.09:50:03.232>
actor = 'cheryl.sabella'
assignee = 'steven.daprano'
closed = True
closed_date = <Date 2019-03-28.09:50:03.233>
closer = 'cheryl.sabella'
components = ['Library (Lib)']
creation = <Date 2009-07-05.15:15:10.511>
creator = 'scott_daniels'
dependencies = []
files = ['14472', '42816']
hgrepos = []
issue_num = 6422
keywords = ['patch']
message_count = 30.0
messages = ['90157', '90245', '90264', '90275', '122955', '123711', '164027', '164070', '164071', '164216', '164217', '168796', '238352', '238354', '238879', '265319', '265322', '265324', '265326', '265360', '265418', '265442', '272140', '272142', '272676', '272704', '338958', '339011', '339024', '339026']
nosy_count = 13.0
nosy_names = ['rhettinger', 'scott_daniels', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'vstinner', 'rbcollins', 'eric.araujo', 'steven.daprano', 'asvetlov', 'tshepang', 'python-dev', 'cheryl.sabella']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue6422'
versions = ['Python 3.6']
Activity
scottdaniels commentedon Jul 5, 2009
timeit.main has a _very_ handy autoranging facility to pick an
appropriate number of repetitions when not specified. The autoranging
code should be lifted to a method on Timer instances (so non-main code
can use it). If number is specified as 0 or None, I would like to use
the results of that autoranging code in Timer.repeat and Timer.timeit.
Patch to follow.
scottdaniels commentedon Jul 7, 2009
I've got the code "working" on trunk2 for my tests.
Should I port to py3K before checking in, and give diffs from there, or
what?
amauryfa commentedon Jul 8, 2009
You can still upload available patches to this tracker.
rhettinger commentedon Jul 8, 2009
I would like to look at this in context of all the other proposed build-
outs to timeit.
rhettinger commentedon Nov 30, 2010
This does not conflict with the other proposed changes to timeit and it is in-line with Guido's desire that to expose useful parts currently buried in the command-line logic.
Amaury, you've shown an interest. Would you like to apply it?
pitrou commentedon Dec 9, 2010
Not sure why you chose 0.11 here. It should probably be 0.2 as in the command-line code.
As for applying the patch, this can't be done before 3.2 is released.
rhettinger commentedon Jun 26, 2012
Looking at this again after more time has passes, I still think exposing autoranging is a good idea but I don't like the patch as it stands. It "infects" the API in a number of places and makes the overall module harder to use and learn.
Ideally, there should be a cleaner interface, or more limited API change, or a separate high level function that can autorange existing functions without changing their API.
Anyone care to propose a cleaner API?
ncoghlan commentedon Jun 26, 2012
In bpo-5442, I proposed leaving the architecture of the module alone, and simply exposing the main module functionality as a high level helper function:
The return value would simply be a (number, results) 2-tuple with the number of iterations per test (which may have been calculated automatically), and then a list of the results. To get "timeit" style behavior, simply set "repeat=1".
ncoghlan commentedon Jun 26, 2012
Oops, that link reference should have been to bpo-5441.
vstinner commentedon Jun 28, 2012
Hi, I wrote recently a similar function because timeit is not reliable by default. Results look random and require to run the same benchmark 3 times or more on the command line.
https://bitbucket.org/haypo/misc/src/tip/python/benchmark.py
By default, the benchmark takes at least 5 measures, one measure should be greater than 100 ms, and the benchmark should not be longer than 1 second. I chose these parameters to get reliable results on microbenchmarks like "abc".encode("utf-8").
The calibration function uses also the precision of the timer. The user may define a minimum time (of one measure) smaller than the timer precision, so the calibration function tries to solve such issue. The calibration computes the number of loops and the number of repetitions.
Look at BenchmarkRunner.calibrate_timer() and BenchmarkRunner.run_benchmark().
https://bitbucket.org/haypo/misc/src/bfacfb9a1224/python/benchmark.py#cl-362
9 remaining items
pitrou commentedon May 11, 2016
https://docs.python.org/3/library/timeit.html#timeit.Timer.repeat
(or, similarly, what timeit's __main__ does: report the minimum of all N runs)
ncoghlan commentedon May 12, 2016
The embedded side-effects were my main concern with Scott's original patch, so Steven's callback-based approach strikes me as a definite improvement. However, the awkwardness of the revised calling code in main does make me wonder whether or not this might be better implemented as a generator rather than as a function accepting a callback:
(Originally I had the "if verbose: print" embedded in a direct loop over t.autorange(), but writing it that way made it immediately clear that the scope of the exception handler was too broad, so I changed it to extract all the results and only then print them)
stevendaprano commentedon May 12, 2016
On Thu, May 12, 2016 at 04:49:59AM +0000, Nick Coghlan wrote:
I thought about a generator too, but then I thought about the *non*
verbose case, where you don't care about the intermediate results, only
the final (number, time_taken) pair.
# function with callback:
number, time_taken = t.autorange()
# generator
number, time_taken = list(t.autorange())[-1]
Which hints that your code snippet is buggy, or at least incomplete:
If verbose is False, you never set number and time_taken. So you need an
else clause:
ncoghlan commentedon May 13, 2016
Good point - given that, +1 from me for the callback based version, especially since exception chaining will still disambiguate failures in the callback from other errors.
stevendaprano commentedon Aug 8, 2016
Nick gave a +1 to my auto-range patch with callback on 2016-05-13, and there's been no negative feedback since. Should I go ahead and check it in for 3.6?
rhettinger commentedon Aug 8, 2016
I think the patch is good to go.
python-dev commentedon Aug 14, 2016
New changeset 424eb46f7f3a by Steven D'Aprano in branch 'default':
bpo-6422 add autorange method to timeit.Timer
https://hg.python.org/cpython/rev/424eb46f7f3a
stevendaprano commentedon Aug 15, 2016
Still to do (coming soon):
timeitandrepeatmethods (and functions) fall backon
autorangeif the number is set to 0 or None.csabella commentedon Mar 27, 2019
Hello Steven,
Were you working on the additional functionality that you mentioned in msg272704 or would that be open for someone else to do? Thanks!
stevendaprano commentedon Mar 28, 2019
Please consider it open. I don't expect it to be difficult, it's just
finding the Round Tuits. Perhaps an easy first issue for someone?
csabella commentedon Mar 28, 2019
Steven,
Thank you. Yes, I was thinking the same thing. But it might be better at this point for that change to have its own ticket, so I'll open a new issue for it.
csabella commentedon Mar 28, 2019
The new ticket is bpo-36461.