Repository navigation
datetime needs an "epoch" method #46988
Description
| BPO | 2736 |
|---|---|
| Nosy | @malemburg, @tim-one, @jribbens, @amauryfa, @mdickinson, @abalkin, @pitrou, @catlee, @vstinner, @bitdancer |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/abalkin'
closed_at = <Date 2011-04-25.17:03:30.218>
created_at = <Date 2008-05-01.21:03:25.299>
labels = ['type-feature', 'docs']
title = 'datetime needs an "epoch" method'
updated_at = <Date 2012-06-08.16:41:09.959>
user = 'https://github.com/tebeka'bugs.python.org fields:
activity = <Date 2012-06-08.16:41:09.959>
actor = 'belopolsky'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2011-04-25.17:03:30.218>
closer = 'belopolsky'
components = ['Documentation']
creation = <Date 2008-05-01.21:03:25.299>
creator = 'tebeka'
dependencies = []
files = ['10251', '10256', '12329', '21565']
hgrepos = []
issue_num = 2736
keywords = ['patch']
message_count = 67.0
messages = ['66045', '66140', '66532', '66539', '66601', '66610', '75723', '75899', '75900', '75902', '75903', '75904', '75912', '76003', '76324', '76327', '76329', '76331', '76332', '76340', '76344', '76345', '76351', '76352', '77650', '77651', '99545', '103875', '106229', '106230', '106249', '106251', '106252', '106254', '106255', '124197', '124203', '124204', '124225', '124230', '124231', '124237', '124245', '124248', '124252', '124255', '124256', '124257', '124259', '132695', '132697', '132818', '132977', '132994', '133008', '133009', '133011', '133037', '133039', '133053', '133056', '133058', '133072', '133207', '133245', '134395', '162533']
nosy_count = 23.0
nosy_names = ['lemburg', 'tim.peters', 'ping', 'jribbens', 'guettli', 'amaury.forgeotdarc', 'mark.dickinson', 'davidfraser', 'belopolsky', 'pitrou', 'andersjm', 'catlee', 'vstinner', 'tomster', 'werneck', 'hodgestar', 'Neil Muller', 'erik.stephens', 'steve.roberts', 'r.david.murray', 'vivanov', 'python-dev', 'Jay.Taylor']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue2736'
versions = ['Python 3.3']
Activity
tebeka commentedon May 1, 2008
If you try to convert datetime objects to seconds since epoch and back
it will not work since the microseconds get lost:
werneck commentedon May 3, 2008
That's expected as mktime is just a thin wrapper over libc mktime() and
it does not expect microseconds. Changing time.mktime doesn't seems an
option, so the best alternative is to implement a method in datetime
type. Is there a real demand for C code implementing this to justify it?
hodgestar commentedon May 10, 2008
Attached a patch which adds a .totimetuple(...) method to
datetime.datetime and tests for it.
The intention is that the dt.totimetuple(...) method is equivalent to:
mktime(dt.timetuple()) + (dt.microsecond / 1000000.0)
hodgestar commentedon May 10, 2008
Patch adding documentation for datetime.totimestamp(...).
tebeka commentedon May 11, 2008
I think the name is not good, should be "toepoch" or something like that.
NeilMuller commentedon May 11, 2008
datetime has fromtimestamp already, so using totimestamp keeps naming
consistency (see toordinal and fromordinal).
vstinner commentedon Nov 11, 2008
See also bpo-1673409
vstinner commentedon Nov 15, 2008
I like the method, but I have some comments about the new method:
instruction
using a block in case of time is not NULL?
PyFloat_FromDouble()
I wrote a similar patch before reading
add-datetime-totimestamp-method.diff which does exactly the same... I
attach my patch but both should be merged.
vstinner commentedon Nov 15, 2008
Here is a merged patch of the three patches. Except the C
implementation of datetime_totimestamp() (written by me), all code is
written by hodgestar.
abalkin commentedon Nov 15, 2008
I would like to voice my opposition the totimestamp method.
Representing time as a float is a really bad idea (originated at
Microsoft as I have heard). In addition to the usual numeric problems
when dealing with the floating point, the resolution of the floating
point timestamp varies from year to year making it impossible to
represent high resolution historical data.
In my opinion both time.time() returning float and
datetime.fromtimestamp() taking a float are both design mistakes and
adding totimestamp that produces a float will further promote a bad
practice.
I would not mind integer based to/from timestamp methods taking and
producing seconds or even (second, microsecond) tuples, but I don't
think changing fromtimestamp behavior is an option.
vstinner commentedon Nov 15, 2008
Le Saturday 15 November 2008 02:15:30 Alexander Belopolsky, vous avez écrit :
It's too late to break the API (Python3 is in RC stage ;-)), but we can create
new methods like:
datetime.fromepoch(seconds, microseconds=0) # (int/long, int)
datetime.toepoch() -> (seconds, microseconds) # (int/long, int)
abalkin commentedon Nov 15, 2008
On Fri, Nov 14, 2008 at 8:37 PM, STINNER Victor <report@bugs.python.org> wrote:
While 1970 is the most popular epoch, I've seen 1900, 2000 and even
2035 (!) being used as well. Similarly, nanoseconds are used in high
resolution time sources at least as often as microseconds. This makes
fromepoch() ambiguous and it is really unnecessary because it can be
written as epoch + timedelta(0, seconds, microseconds).
I would much rather have divmod implemented as you suggested in
bpo-2706 . Then toepoch is simply
vstinner commentedon Nov 15, 2008
Le Saturday 15 November 2008 04:17:50 Alexander Belopolsky, vous avez écrit :
I tried yesterday and it doesn't work!
We lost an hour durint the 1st World War :-)
Whereas my implementation using mktime() works:
-1670000000.0
47 remaining items
malemburg commentedon Apr 5, 2011
STINNER Victor wrote:
Sorry, but no. I'm not really a fan of the datetime module and
try to stay away from it whenever I can :-)
Note that dealing with DST in timezones other than the local time
zone, is bound to go wrong without direct access to the tz library.
The C lib doesn't provide any good way to access timezone
information other than the local timezone or UTC.
When dealing with date/time values, it is usually best to stay with
UTC and only transform those values into local times in user
interfaces on the front-end client.
Consequently, I'd suggest to only allow UTC and local timezone
conversions for the method in the datetime module.
abalkin commentedon Apr 5, 2011
On Tue, Apr 5, 2011 at 4:33 AM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
..
Yes, mxDateTime's gmticks()/ticks() pair of functions present a much
more mature design than anything that has been proposed here. It is
telling, however, that no one has mentioned mxDateTime's gmticks() on
this issue in four years. On a duplicate bpo-1673409, Marc did
bring it up, but as far as I can tell, no one responded. See
msg75411.
Google code search,
http://www.google.com/codesearch?hl=en&sa=N&q=gmticks+lang:python
returns only 13 hits for "gmticks". In several instances, the
resulting float is immediately converted to int, in other instances
"gmticks" is mentioned in comments and the code works around its bugs.
I would not use Google Code search as an ultimate arbiter on the
popularity of a feature, so I would really like to hear from the
proponents about real life uses of gmticks() or any other examples
where a similar method "has been reimplemented over and over again."
Well, search for gmticks does not return too many hits outside of
mxDateTime code and manuals, but I had no trouble finding this:
"""
okay, all the MySQLdb dataobject tick-based time handling methods are
broken in various ways -- reconstruct GMT ticks from time module's
mktime...
"""
http://viewvc.tigris.org/ds/viewMessage.do?dsForumId=4251&dsMessageId=656863
Follow the link for some more colorful language describing developer's
experience with the feature.
Note that it is likely that the bug MySQLdb developer complained about
was fixed in mxDateTime at some point,
<http://www.egenix.com/www2002/python/mxDateTime-History.html\>, but
this shows that implementing gmticks() correctly is not as trivial as
those who never tried might think.
Which in turn is a mistake IMO. Note that POSIX does not use float
timestamps for a reason.
This is not the goal that I would support. I would rather see code
that uses datetime module not require time module methods at all.
Victor reported implementing two methods, one to return a float and
another to return a tuple. See msg124259. I am not sure I've seen
that code.
No, it is not about "only one catch". Victor's patch is simply wrong.
For an aware datetime instance it extracts DST flag from tzinfo, but
ignores the offset.
abalkin commentedon Apr 5, 2011
MAL> Since most of the datetime module was inspired by mxDateTime,
MAL> I wonder why [ticks()/gmticks()] were left out. (msg75411)
"""
The datetime module intended to be an island of relative sanity.
Because the range of dates "timestamps" can represent varies across
platforms (and even "the epoch" varies), datetime doesn't even try to
produce timestamps directly -- datetime is more of an alternative to
"seconds from the epoch" schemes. Because datetime objects have
greater range and precision than timestamps, conversion is
problem-free in only one direction. It's not a coincidence that
that's the only direction datetime supplies ;-)
""" - Tim Peters
http://bytes.com/topic/python/answers/522572-datetime-timestamp
I will also add that fromtimestamp() is not invertible in the presence of DST. That's why mxDatetime.ticks() takes a DST flag making it effectively a multi-valued function. Note that naive users, who just want to pass datetime values to an OS function expecting a float, most likely will not have means of properly obtaining DST flag.
malemburg commentedon Apr 5, 2011
Alexander Belopolsky wrote:
mxDateTime needs those two methods, since it doesn't natively
use timezones. The .ticks() method is used for local time values,
.gmticks() for UTC ones; that's why there are two methods.
The .gmticks() method is always used when storing UTC values
in mxDateTime instances, which actually is the preferred way
of storing data in databases. Google Code doesn't really count
much, since it only scans a limited number of OSS code bases.
Most of our users are commercial users who use the tools
in-house.
Note that implementing .gmticks() is fairly easy on POSIX
conform systems. On most others, timegm() can be used. If
that doesn't exist, things get tricky, but that case should
be rare nowadays.
Note that he was referring to the .ticks() method, not the .gmticks()
method. The patch doesn't say which version of mxDateTime he was
using. The bug mentioned in the changelog was fixed in 1998. It is
possible, however, that the mktime() on his system was broken - which
is why I added a test for it in mxDateTime.
The time module is our reference in this case and this tries hard
to add fractions of a second to the value :-)
Note that sub-second accuracy relies on a number of factors,
the storage format most certainly is the least important
aspect ;-)
On many systems, you only get 1/100s accuracy, on others, the timer
ticks in fixed increments, giving you even weirder sub-second values
(e.g. time appears to stay constant between time.time() calls).
OTOH, there's a new set of APIs for nano-second accuracy available
now, which the datetime module objects cannot represent at all due
to the integer-based storage format.
BTW: The integer format was chose in order to keep the memory
footprint of the objects low.
No chance :-) In practice, the time module gets used a lot
for date/time storage or to quickly measure time deltas.
Some people also prefer time module ticks due to their lower
memory footprint, esp. when it comes to storing thousands of
time values in time series.
I had a look at the last patch on this ticket.
True, so make that two issues ;-)
malemburg commentedon Apr 5, 2011
Alexander Belopolsky wrote:
IMHO, the whole concept of DST is broken, but that's not our fault :-)
Ditching the concept just because it is known to fail for
one hour out of 8760 you have in a typical year doesn't really
warrant breaking the "practicality beats purity" guideline.
Otherwise, we'd have to ditch the date support in the datetime module
too: after all, Feb 29 only exists every 4 years (well, most of the
time) - and that's one day out of 1461 in those 4 years, so an
even worse ratio :-)
And I'm not even starting to talk about ditching the concept
of Unix ticks to begin with, as a result of having leap seconds
causing POSIX ticks values not matching (real) UTC ticks.
In reality, all these things hardly ever matter and if they do,
users will either know that they have to make conscious decision,
simply don't care or decide not to care.
BTW: A "timestamp" usually refers to the combination of date and
time. The time.time() return value is "seconds since the Epoch".
I usually call those values "ticks" (not sure whether it's standard
term of not, but always writing "seconds since Epoch" wasn't an
option either ;-)).
Date/time is fun, isn't it ?
abalkin commentedon Apr 5, 2011
Let me state my position on this issue once again. Converting datetime values to float is easy. If your dt is a naive instance representing UTC time:
If your dt is an aware instance:
These recipes are easy to adjust for your application needs. One application may want millisecond or microsecond ticks, another might want to carry subsecond presision in a separate integer, third may want to avoid timestamps before 1970 or after 2038 or ignore microseconds altogether. No matter what a hypothetical datetime.epoch() will provide, most of applications will need to add a line or two to its code to serve their needs. Applications that will use dt.epoch() naively without thinking what dt represents (say local or UTC) will be buggy.
The only related feature that I think is missing from datetime module is the ability to obtain local time as an aware datetime instance and to convert a naive datetime instance assumed to represent local time to an aware one.
This is the subject of bpo-9527, but there is a resistance to adding that feature.
abalkin commentedon Apr 5, 2011
On Tue, Apr 5, 2011 at 1:45 PM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
..
In Unix context, the term "timestamp" is usually associated with the
various time values that OS stores with the files. I think this use
is due to the analogy with physical "received on" timestamps used on
paper documents. Since it is well-known that Unix filesystems store
time values as seconds since Epoch, it is common to refer to these
values as "Unix timestamps".
See, for example:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/touch.html
vivanov commentedon Apr 7, 2011
On 04/05/2011 18:22, Alexander Belopolsky wrote:
Refusing to cooperate with the rest of the world is not sane by my books.
On 04/05/2011 21:06, Alexander Belopolsky wrote:
Please add these lines to the datetime module's documentation. In some
central, well lit place. I believe that if nothing else, the whole
discussion should have proved to you that there are many people looking
for them.
OTOH a sinceepoch(epoch=datetime(1970,1,1)) method of the datetime class
should be equally easy. Would be especially useful if few of the more
frequently used EPOCHs are provided as constants.
abalkin commentedon Apr 7, 2011
On Thu, Apr 7, 2011 at 6:20 AM, Velko Ivanov <report@bugs.python.org> wrote:
..
This is precisely what I suggested at the end of msg132697 above. See
attached patch (bpo-2736-doc.diff) for a proposed documentation
enhancement.
python-dev commentedon Apr 25, 2011
New changeset b55eac85e39c by Alexander Belopolsky in branch 'default':
Issue bpo-2736: Documented how to compute seconds since epoch.
http://hg.python.org/cpython/rev/b55eac85e39c
python-dev commentedon Jun 8, 2012
New changeset 6671c5039e15 by Alexander Belopolsky in branch 'default':
Issue bpo-2736: Added datetime.timestamp() method.
http://hg.python.org/cpython/rev/6671c5039e15