timeit called from within Python should allow autoranging

@rhettinger

BPO	6422
Nosy	@rhettinger, @amauryfa, @ncoghlan, @pitrou, @vstinner, @rbtcollins, @merwok, @stevendaprano, @asvetlov, @csabella
Files	timeit.patch: Patch including changes to timeit, test, and doc timeit-autorange.patch: minimal proof-of-concept patch to add autorange method to Timer

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/stevendaprano'
closed_at = <Date 2019-03-28.09:50:03.233>
created_at = <Date 2009-07-05.15:15:10.511>
labels = ['type-feature', 'library']
title = 'timeit called from within Python should allow autoranging'
updated_at = <Date 2019-03-28.09:50:03.232>
user = 'https://bugs.python.org/scottdaniels'

bugs.python.org fields:

activity = <Date 2019-03-28.09:50:03.232>
actor = 'cheryl.sabella'
assignee = 'steven.daprano'
closed = True
closed_date = <Date 2019-03-28.09:50:03.233>
closer = 'cheryl.sabella'
components = ['Library (Lib)']
creation = <Date 2009-07-05.15:15:10.511>
creator = 'scott_daniels'
dependencies = []
files = ['14472', '42816']
hgrepos = []
issue_num = 6422
keywords = ['patch']
message_count = 30.0
messages = ['90157', '90245', '90264', '90275', '122955', '123711', '164027', '164070', '164071', '164216', '164217', '168796', '238352', '238354', '238879', '265319', '265322', '265324', '265326', '265360', '265418', '265442', '272140', '272142', '272676', '272704', '338958', '339011', '339024', '339026']
nosy_count = 13.0
nosy_names = ['rhettinger', 'scott_daniels', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'vstinner', 'rbcollins', 'eric.araujo', 'steven.daprano', 'asvetlov', 'tshepang', 'python-dev', 'cheryl.sabella']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue6422'
versions = ['Python 3.6']

The calibration function uses also the precision of the timer.

Oh, I forgot to mention that it computes the precision in Python, it doesn't read the precision announced by the OS or the precision of the C structure.
https://bitbucket.org/haypo/misc/src/bfacfb9a1224/python/benchmark.py#cl-66

In bpo-5442, I proposed leaving the architecture of the module alone, and
simply exposing the main module functionality as a high level helper
function:

Agreed with Nick's approach above.

Victor, if you want to improve timeit's reliability, please open a separate issue.

I'm confused by the feedback on the patch. It adds a single new function, doesn't alter the public interface for any existing functions, and seems fit for purpose. Could someone help me understand how its deficient?

Filed bpo-23693 for the accuracy thing.

The current patch moves print operations inside timeit() and repeat(), instead of leaving the main() function as the only one with side effects.

My counter-proposal was to instead extract the current main functionality out into a side-effect free public API of its own, and change the existing main function to call that new API and print the results.

This issue seems to have lost momentum, I'd like to revive it by proposing a slightly different interface for the autorange function.

Attached is a proof-of-concept patch. I've moved the code which determines the number of loops out of the main function into a new Timer method, autorange. The main difference between my approach and Scott's is that my autorange method doesn't do any printing directly, it takes an optional callback function. This lets the caller take responsibility for all output.

If this approach is acceptable, I hope to:

(in 3.6) add tests and docs for the new method;
(in 3.6 if time permits, otherwise 3.7) modify the timeit and repeat methods and functions so that they can optionally call autorange(), e.g. if the caller passes 0 as the number.

I'm not sure that there's a good reason to add a top-level autorange() function to match the timeit() and repeat() functions. Especially not once they gain the ability to autorange themselves.

I think my approach will be compatible with cleaning up and refactoring the main() function. At the moment, main() is a mess IMO, it handles argument processing, autoranging, units of time, and unreliable timing detection all from one function.

I would suggest making the 0.2 tunable as an optional argument. Different applications (benchmarks) may want different duration / precision tradeoffs.
I also notice the repeat functionality isn't included in the patch, is there a reason?

I would suggest making the 0.2 tunable as an optional argument.

Sounds like a good idea to me.

I also notice the repeat functionality isn't included in the patch, is there a reason?

I don't understand which repeat functionality you're referring to.

I don't understand which repeat functionality you're referring to.

https://docs.python.org/3/library/timeit.html#timeit.Timer.repeat

(or, similarly, what timeit's __main__ does: report the minimum of all N runs)

The embedded side-effects were my main concern with Scott's original patch, so Steven's callback-based approach strikes me as a definite improvement. However, the awkwardness of the revised calling code in main does make me wonder whether or not this might be better implemented as a generator rather than as a function accepting a callback:

try:
    results = list(t.autorange())
except:
    t.print_exc()
    return 1
if verbose:
    for number, time_taken in results:
        msg = "{} loops -> {:.{}g} secs"
        print(msg.format(number, time_taken, precision))

(Originally I had the "if verbose: print" embedded in a direct loop over t.autorange(), but writing it that way made it immediately clear that the scope of the exception handler was too broad, so I changed it to extract all the results and only then print them)

On Thu, May 12, 2016 at 04:49:59AM +0000, Nick Coghlan wrote:

The embedded side-effects were my main concern with Scott's original
patch, so Steven's callback-based approach strikes me as a definite
improvement. However, the awkwardness of the revised calling code in
main does make me wonder whether or not this might be better
implemented as a generator rather than as a function accepting a
callback:

I thought about a generator too, but then I thought about the *non*
verbose case, where you don't care about the intermediate results, only
the final (number, time_taken) pair.

# function with callback:
number, time_taken = t.autorange()

# generator
number, time_taken = list(t.autorange())[-1]

Which hints that your code snippet is buggy, or at least incomplete:

try:
    results = list(t.autorange())
except:
    t.print_exc()
    return 1
if verbose:
    for number, time_taken in results:
        msg = "{} loops -\> {:.{}g} secs"
        print(msg.format(number, time_taken, precision))

If verbose is False, you never set number and time_taken. So you need an
else clause:

else:
    number, time_taken = results[-1]

Good point - given that, +1 from me for the callback based version, especially since exception chaining will still disambiguate failures in the callback from other errors.

Nick gave a +1 to my auto-range patch with callback on 2016-05-13, and there's been no negative feedback since. Should I go ahead and check it in for 3.6?

I think the patch is good to go.

New changeset 424eb46f7f3a by Steven D'Aprano in branch 'default':
bpo-6422 add autorange method to timeit.Timer
https://hg.python.org/cpython/rev/424eb46f7f3a

Still to do (coming soon):

make the 0.2s time configurable;
have timeit and repeat methods (and functions) fall back
on autorange if the number is set to 0 or None.

Hello Steven,

Were you working on the additional functionality that you mentioned in msg272704 or would that be open for someone else to do? Thanks!

Were you working on the additional functionality that you mentioned in
msg272704 or would that be open for someone else to do? Thanks!

Please consider it open. I don't expect it to be difficult, it's just
finding the Round Tuits. Perhaps an easy first issue for someone?

Steven,

Thank you. Yes, I was thinking the same thing. But it might be better at this point for that change to have its own ticket, so I'll open a new issue for it.

The new ticket is bpo-36461.

Navigation Menu

Repository navigation

timeit called from within Python should allow autoranging #50671

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

timeit called from within Python should allow autoranging #50671

Description

Activity

vstinner commented on Jun 28, 2012

pitrou commented on Aug 21, 2012

rbtcollins commented on Mar 17, 2015

rbtcollins commented on Mar 17, 2015

ncoghlan commented on Mar 22, 2015

stevendaprano commented on May 11, 2016

pitrou commented on May 11, 2016

stevendaprano commented on May 11, 2016

pitrou commented on May 11, 2016

ncoghlan commented on May 12, 2016

stevendaprano commented on May 12, 2016

ncoghlan commented on May 13, 2016

stevendaprano commented on Aug 8, 2016

rhettinger commented on Aug 8, 2016

python-dev commented on Aug 14, 2016

stevendaprano commented on Aug 15, 2016

csabella commented on Mar 27, 2019

stevendaprano commented on Mar 28, 2019

csabella commented on Mar 28, 2019

csabella commented on Mar 28, 2019

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions