PEP 410: Use decimal.Decimal type for timestamps

@loewis

BPO	13882
Nosy	@loewis, @mdickinson, @abalkin, @pitrou, @vstinner, @ericvsmith, @skrah
Files	time_decimal-18.patch timestamp_datetime-2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-03-02.21:38:35.975>
created_at = <Date 2012-01-26.21:00:37.196>
labels = ['library']
title = 'PEP 410: Use decimal.Decimal type for timestamps'
updated_at = <Date 2012-03-02.21:38:35.974>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2012-03-02.21:38:35.974>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2012-03-02.21:38:35.975>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2012-01-26.21:00:37.196>
creator = 'vstinner'
dependencies = []
files = ['24538', '24539']
hgrepos = []
issue_num = 13882
keywords = ['patch']
message_count = 38.0
messages = ['152031', '152032', '152035', '152129', '152131', '152133', '152134', '152290', '152303', '152304', '152353', '152356', '152358', '152359', '152374', '152417', '152456', '152489', '152490', '152491', '152502', '152571', '152795', '152796', '152818', '152823', '152826', '152835', '152838', '152916', '153007', '153197', '153199', '153201', '153240', '153520', '153539', '154801']
nosy_count = 9.0
nosy_names = ['loewis', 'mark.dickinson', 'belopolsky', 'pitrou', 'vstinner', 'eric.smith', 'Arfrever', 'skrah', 'Alexander.Belopolsky']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue13882'
versions = ['Python 3.3']

Attached patch adds an optional format argument to time.time(), time.clock(), time.wallclock(), time.clock_gettime() and time.clock_getres() to get the timestamp as a different format. By default, the float type is still used, but it will be possible to pass "format" to get the timestamp as a decimal.Decimal object.

Some advantages of using decimal.Decimal instead of float:

no loss of precision during conversion from base 2 to base 10 (converting the float to a string)
the resolution of the clock is stored in the Decimal object
for big number, Decimal doesn't loose precision

Using Decimal is also motivated by the fact than Python 3.3 has now access to clock having a resolution of 1 nanosecond: clock_gettime(CLOCK_REALTIME). Well, it doesn't mean that the clock is accurate, but Python should not loose precision just because it uses floatting point instead of integers.

About the API: I chose to add a string argument to allow maybe later to support user defined formats, or at least add new builtin formats. For example, someone proposed 128 bits float in the issue bpo-11457.

--

The internal format is:

typedef struct {
    time_t seconds;
    /* floatpart can be zero */
    size_t floatpart;
    /* divisor cannot be zero */
    size_t divisor;
    /* log10 of the clock resoltion, the real resolution is 10^resolution,
       resolution is in range [-9; 0] */
    int resolution;
} _PyTime_t;

I don't know if size_t is big enough to store any "floatpart" value: time.clock() uses a LARGE_INTEGER internally. I tested my patch on Linux 32 and 64 bits, but not yet on Windows.

The internal function encoding a timestamp to Decimal caches the resolution objects (10^resolution, common clock resolutions: 10^-6, 10^-9) in a dict, and a Context object.

I computed that 22 decimal digits should be enough to compute a timestamp of 10,000 years. Extract of my patch:

"Use 12 decimal digits to store 10,000 years in seconds + 9 decimal digits for the floating part in nanoseconds + 1 decimal digit to round correctly"

>>> str(int(3600*24*365.25*10000))
'315576000000'
>>> len(str(int(3600*24*365.25*10000)))
12

--

See also the issue bpo-11457 which is linked to this topic, but not exactly the same because it concerns a low level function (os.stat()).

Some examples of the API:

$ ./python 
Python 3.3.0a0 (default:52f68c95e025+, Jan 26 2012, 21:54:31) 
>>> import time
>>> time.time()
1327611705.948446
>>> time.time('decimal')
Decimal('1327611708.988419')
>>> t1=time.time('decimal'); t2=time.time('decimal'); t2-t1
Decimal('0.000550')
>>> t1=time.time('float'); t2=time.time('float'); t2-t1
5.9604644775390625e-06
>>> time.clock_gettime(time.CLOCK_MONOTONIC, 'decimal')
Decimal('1211833.389740312')
>>> time.clock_getres(time.CLOCK_MONOTONIC, 'decimal')
Decimal('1E-9')
>>> time.clock()
0.12
>>> time.clock('decimal')
Decimal('0.120000')

Windows code (win32_clock) was wrong in time_decimal-2.patch: it is fixed in patch version 3.

Some tests on Windows made me realize that time.time() has a resolution of 1 millisecond (10^-3) and not of a microsecond (10^-6) on Windows! It is time to use GetSystemTimeAsFileTime! => see issue bpo-13845.

Can we pick an API for this functionality that does not follow the worst of design anti-patterns? Constant arguments, varying return type, hidden import, and the list can go on.

What is wrong with simply creating a new module, say "hirestime" with functions called decimal_time(), float_time(), datetime_time() and whatever else you would like. Let's keep the good old 'time' module simple.

Well, creating a separate module is an anti-pattern in itself. calendar vs. time vs. datetime, anyone?
I would instead propose separate functions: decimal_time, decimal_clock... or, if you prefer, time_decimal and so on.

On Fri, Jan 27, 2012 at 5:17 PM, Antoine Pitrou <report@bugs.python.org> wrote:

Well, creating a separate module is an anti-pattern in itself. calendar vs. time vs. datetime, anyone?

Are you serious? Since the invention of structural programming,
creating a separate module for distinct functionality has been one of
the most powerful design techniques. If I recall correctly, most of
the original GoF patterns were about separating functionality into a
separate module or a separate class. The calendar module is indeed a
historical odd-ball, but what is wrong with time and datetime?

Are you serious? Since the invention of structural programming,
creating a separate module for distinct functionality has been one of
the most powerful design techniques.

Yes, I'm serious, and I don't see what structural programming or design
patterns have to do with it.

And we're not even talking about "distinct functionality", we're talking
about the exact same functionality, except that the return type has
slightly different implementation characteristics. Doing a module split
is foolish.

Constant arguments

What do you call a constant argument? "float" and "decimal"? You would prefer a constant like time.FLOAT_FORMAT? Or maybe a boolean (decimal=True)?

I chose a string because my first idea was to add a registry to support other format, maybe user defined formats, like the one used by Unicode codecs.

If we choose to not support other formats, but only float and decimal, a simpler API can be designed.

Another possible format would be "tuple": (intpart: int, floatpart: int, divisor: int), a low level type used to "implement" other user-defined types. Using such tuple, you have all information (clock value and clock resolution) without losing information.

varying return type

I agree that it is something uncommon in Python. I know os.listdir(bytes)->bytes and os.listdir(str)->str. I suppose that there are other functions with a different result type depending on the input.

I am not attached to my API, it was just a proposition.

hidden import

Ah? I wouldn't call it hidden because I don't see how a function can return a decimal.Decimal object without importing it. If you consider that it is surprising (unexepected), it can be documented.

and the list can go on.

What else?

What is wrong with simply creating a new module, say "hirestime"
with functions called decimal_time(), float_time(), datetime_time()
and whatever else you would like.

Hum, adding a new module would need to duplicate code. The idea of adding an argument is also to simplify the implementation: most code is shared. We can still share a lot of code if we choose to add a new function in th time module instead of adding a new argument to existing functions.

Let's keep the good old 'time' module simple.

What is complex in my patch? It doesn't break backward compatibility and should have a low (or null) overhead in runtime speed if the format is not set.

--

I notified something surprising in my patch: "t1=time.time("decimal"); t2=time.time("decimal"); t2-t1" returns something bigger than 20 ms... That's because the "import decimal" is done after reading the first clock value, and not before.

Patch version 4, minor update:

remove the resolution field of _PyTime_t and remove int_log10(): use 1/divisior as the resolution to support divisor different than a power of 10 (e.g. the cpu frequency on Windows)
inline and remove _PyTime_FromTimespec()

Another possible format would be "tuple"

Or, I forgot an obvious format: "datetime"!

Version 5:

add "datetime" and "timespec" formats: datetime.datetime object and (sec: int, nsec: int)
add timestamp optional format to os.stat(), os.lstat(), os.fstat(), os.fstatat()
support passing the timestamp format as a keyword: time.time(format="decimal")

I am not really conviced by the usefulness of "timespec" format, but it was just an example for bpo-11457.

The "datetime" format is surprising for time.clock() and time.wallclock(), these timestamps use an arbitrary start. I suppose that time.clock(format="datetime") and time.wallclock(format="datetime") should raise a ValueError.

On Sun, Jan 29, 2012 at 6:42 PM, STINNER Victor <report@bugs.python.org> wrote:
..

What do you call a constant argument? "float" and "decimal"?
You would prefer a constant like time.FLOAT_FORMAT?
Or maybe a boolean (decimal=True)?

Yes. This was explained on python-dev not so long ago:

http://mail.python.org/pipermail/python-dev/2010-July/102100.html

The problem is not with the type of the argument (although boolean or
enum argument type is often a symptom pointing to the issue.) The
issue is that an argument that is never given a variable value at the
call site is usually a sign of an awkward API. For example, what
would you prefer:

math('log', x)

or

math.log(x)

?

I chose a string because my first idea was to add a registry to support other format,
maybe user defined formats, like the one used by Unicode codecs.

With all my respect for MAL, codecs are not my favorite part of the
python library.

One possibility (still awkward IMO) would be to use the return type as
the format specifier. This would at least require the user to import
datetime or decimal before calling time() with corresponding format.
Few users would tolerate I/O delay when they want to get time with
nanosecond precision.

os.stat().st_birthtime should depend on the timestamp argument.

A timestamp optional argument should also be added to os.wait3() and os.wait4() for the utime and stime fields of the rusage tuple.

I created the issue bpo-13964 to cleanup the API of os.*utime*() functions.

fill_time() should use denominator=1 if the OS doesn't support timestamp with a subsecond resolution. See also issue bpo-13964.

Patch version 12:

os.stat().st_birthtime uses also the timestamp argument
Add an optional timestamp argument to os.wait3() and os.wait4(): change type of utime and stime attributes of the resource usage
os.stat() changes the timestamp resolution depending if nanosecond resolution is available or not

I realized that resource.getrusage() should also be modified. I will maybe do that in another version of the patch, or maybe change resource usage in another patch.

Patch version 13:

os.utime(path) sets the access and modification time using the currenet time with a subsecond resolution (e.g. microsecond resolution on Linux)
os.*utime*() functions uses _PyTime_t type and functions
add many functions to manipulate timeval, timespec and FILETIME types with _PyTime_t, add _PyTime_SetDenominator() function for that
coding style: follow PEP-7 rules for braces

So more functions (including os.*utime*()) "accept" Decimal, but using an implicit conversion to float.

Patch version 14:

rewrite the conversion from float to _PyTime_t: use base 2 with high precision and then simplify the fraction. The conversion from decimal.Decimal uses base 10 and do also simplify the fraction.
write tests on functions converting _PyTime_t using _testcapi
add timestamp argument to signal.getitimer(), signal.setitimer(), resource.getrusage()
signal.sigtimedwait() uses _PyTime_t and now expects a number and no more a tuple (function added to Python 3.3, so no backward compatibility issue). See also the issue bpo-13964
time.sleep() uses _PyTime_t. See also the issue bpo-13981 (use nanosleep())
datetime.datetime.now() and datetime.datetime.utcnow() uses _PyTime_t
catch integer overflow on _PyTime_AsTimeval(), _PyTime_AsTimespec() and more

This patch gives you an overview of the whole PEP-410 implementation, but it should not be applied in one shot. It would be better to commit it step by step:

add _PyTime_t API
use _PyTime_t for the time module
use _PyTime_t for the os module
use _PyTime_t for the more modules
etc.

We can start by adding the API and use it in the time module, and then rediscuss changes on other modules.

Patch version 15:

round "correctly"
datetime.date.fromtimestamp() and datetime.datetime.fromtimestamp() reuses _PyTime_t API to support decimal.Decimal without loss of precision
add more tests

(Oops, I attached the wrong patch.)

New try, set the version to 16 to avoid the confusion.

test_time is failing on Windows.

Here is the version 17 of my patch. This version is mostly complete and so can be reviewed. Summary of the patch:

Add a _PyTime_t structure to store any timestamp in any resolution, universal structure used by all functions manipulating timestamps instead of C double to avoid loss of precision
Add many functions to create timestamp (set _PyTime_t structure) or to get a timestamp in a specific format (int, float, Decimal, timeval or timespec structure, in milliseconds, etc.)
Round to nearest with ties going away from zero (rounding method called "ROUND_HALF_UP" in Decimal)
Functions creating timestamps get a new optional timestamp argument to specify the requested return type, e.g. time.time(timestamp=int) returns an int
Functions getting timestamps argument now also support decimal.Decimal
Raise an OverflowError instead of a ValueError if a timestamp cannot be stored in a C time_t type

The patch is huge, but as I wrote before, I will split it into smaller parts:

Add _PyTime_t API
Use the new API in the time module
Use the new API in the os module
etc.

Changes in the version 17 of my patch:

tested on Linux 32/64 bits, OpenBSD 64 bits, FreeBSD 64 bits, Windows 64 bits
fix portability issues (for various time_t and C long sizes)

Patch version 18:

Fix a loss of precision in _PyTime_SetDenominator()
Add more tests on integer overflow

I also updated the patch adding datetime.datetime support because some people are interested by the type, even I don't think that it is interesting to add it. datetime.datetime is only usable with time.time(), os.*stat() and time.clock_gettime(), whereas it is incompatible with all other functions.

TODO:

the conversion from Decimal to _PyTime_t does still use a cast to
float and so lose precision
the PEP must be accepted :-)

The PEP has been rejected, so I close the issue.

Navigation Menu

Repository navigation

PEP 410: Use decimal.Decimal type for timestamps #58090

14 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

PEP 410: Use decimal.Decimal type for timestamps #58090

Description

Activity

vstinner commented on Jan 26, 2012

vstinner commented on Jan 26, 2012

vstinner commented on Jan 26, 2012

abalkin commented on Jan 27, 2012

pitrou commented on Jan 27, 2012

AlexanderBelopolsky commented on Jan 27, 2012

pitrou commented on Jan 27, 2012

vstinner commented on Jan 29, 2012

vstinner commented on Jan 30, 2012

vstinner commented on Jan 30, 2012

vstinner commented on Jan 30, 2012

AlexanderBelopolsky commented on Jan 30, 2012

14 remaining items

vstinner commented on Feb 8, 2012

vstinner commented on Feb 8, 2012

vstinner commented on Feb 8, 2012

vstinner commented on Feb 8, 2012

vstinner commented on Feb 8, 2012

vstinner commented on Feb 10, 2012

vstinner commented on Feb 12, 2012

vstinner commented on Feb 12, 2012

vstinner commented on Feb 12, 2012

vstinner commented on Feb 13, 2012

vstinner commented on Feb 17, 2012

vstinner commented on Feb 17, 2012

vstinner commented on Mar 2, 2012

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions