Repository navigation
datetime needs an "epoch" method #46988
Description
| BPO | 2736 |
|---|---|
| Nosy | @malemburg, @tim-one, @jribbens, @amauryfa, @mdickinson, @abalkin, @pitrou, @catlee, @vstinner, @bitdancer |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/abalkin'
closed_at = <Date 2011-04-25.17:03:30.218>
created_at = <Date 2008-05-01.21:03:25.299>
labels = ['type-feature', 'docs']
title = 'datetime needs an "epoch" method'
updated_at = <Date 2012-06-08.16:41:09.959>
user = 'https://github.com/tebeka'bugs.python.org fields:
activity = <Date 2012-06-08.16:41:09.959>
actor = 'belopolsky'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2011-04-25.17:03:30.218>
closer = 'belopolsky'
components = ['Documentation']
creation = <Date 2008-05-01.21:03:25.299>
creator = 'tebeka'
dependencies = []
files = ['10251', '10256', '12329', '21565']
hgrepos = []
issue_num = 2736
keywords = ['patch']
message_count = 67.0
messages = ['66045', '66140', '66532', '66539', '66601', '66610', '75723', '75899', '75900', '75902', '75903', '75904', '75912', '76003', '76324', '76327', '76329', '76331', '76332', '76340', '76344', '76345', '76351', '76352', '77650', '77651', '99545', '103875', '106229', '106230', '106249', '106251', '106252', '106254', '106255', '124197', '124203', '124204', '124225', '124230', '124231', '124237', '124245', '124248', '124252', '124255', '124256', '124257', '124259', '132695', '132697', '132818', '132977', '132994', '133008', '133009', '133011', '133037', '133039', '133053', '133056', '133058', '133072', '133207', '133245', '134395', '162533']
nosy_count = 23.0
nosy_names = ['lemburg', 'tim.peters', 'ping', 'jribbens', 'guettli', 'amaury.forgeotdarc', 'mark.dickinson', 'davidfraser', 'belopolsky', 'pitrou', 'andersjm', 'catlee', 'vstinner', 'tomster', 'werneck', 'hodgestar', 'Neil Muller', 'erik.stephens', 'steve.roberts', 'r.david.murray', 'vivanov', 'python-dev', 'Jay.Taylor']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue2736'
versions = ['Python 3.3']
Activity
andersjm commentedon Nov 18, 2008
Any thoughts to time zone/DST handling for naive datetime objects? E.g.
suppose the datetime object was created by .utcnow or .utcfromtimestamp.
For aware datetime objects, I think the time.mktime(dt.timetuple())
approach doesn't work; the tz info is lost in the conversion to time tuple.
davidfraser commentedon Nov 24, 2008
----- "Alexander Belopolsky" <report@bugs.python.org> wrote:
The point for me is that having to interact with Microsoft systems that require times means that the conversions have to be done. Is it better to have everybody re-implement this, with their own bugs, or to have a standard implementation? I think it's clearly better to have it as a method on the object. Of course, we should put docs in describing the pitfalls of this approach...
abalkin commentedon Nov 24, 2008
On Mon, Nov 24, 2008 at 9:04 AM, David Fraser <report@bugs.python.org> wrote:
...
I did not see the "epoch" proposal as an interoperability with
Microsoft systems feature. If this is the goal, a deeper analysis of
the Microsoft standards is in order. For example, what is the valid
range of the floating point timestamp? What is the range for which
fromepoch (float to datetime) translation is valid? For example, if
all floats are valid timestamps, then fromepoch can be limited to +/-
2**31 or to a smaller range where a float has enough precision to
roundtrip microseconds.
As far as I know, interoperability with Microsoft systems requires
re-implementation of their bugs many of which are not documented. For
example, OOXML requires that 1900 be treated as a leap year at least
in some cases. When you write your own implementation, at least you
have the source code to your own bugs.
Yes, having a well documented high resolution "time since epoch" to
"local datetime" method in the datetime module is helpful if
non-trivial timezones (such as the one Victor lives in) are supported.
However, introducing floating point pitfalls into the already
overcomplicated realm of calendar calculations would be a mistake.
I believe the correct approach would be to extend fromtimestamp (and
utcfromtimestamp) to accept a (seconds, microseconds) tuple as an
alternative (and in addition) to the float timestamp. Then
totimestamp can be implemented to return such tuple that
fromtimestamp(totimestamp(dt) == dt for any datetime dt and
totimestamp(fromtimestamp((s,us))) == (s, us) for any s and us within
datetime valid range (note that s will have to be a long integer to
achieve that).
In addition exposing the system gettimeofday in the time module to
produce (s, us) tuples may be helpful to move away from float
timestamps produced by time.time(), but with totimestamp as proposed
above that would be equivalent to datetime.now().totimestamp().
vstinner commentedon Nov 24, 2008
About the timestamp, there are many formats:
(a) UNIX: 32 bits signed integer, number of seconds since the 1st
january 1970.
compiled python script header (.pyc/.pyo)
(b) UNIX64: 64 bits signed integer, number of seconds since the 1st
january 1970
(c) UNIX: 32 bits unsigned integer, number of seconds since the 1st
january 1904
(d) UUID60: 60 bits unsigned integer, number of 1/10 microseconds
since the 15st october 1582
Microsoft world)
(e) Win64: 64 bits unsigned integer, number of 1/10 microseconds since
the 1st january 1601
video (.asf), Windows link (.lnk)
(f) MSDOS DateTime or TimeDate: bitfield with 16 bits for the date and
16 bits for the time. Time precision is 2 seconds, year is in range
[1980; 2107]
(.doc), ACE archive (.ace), ZIP archive (.zip), RAR achive (.rar)
vstinner commentedon Nov 24, 2008
Timedelta formats:
(a) Win64: 64 bits unsigned integer, number of 1/10 microsecond
(b) 64 bits float, number of seconds
Other file formats use multiple numbers to store a duration:
[AVI video]
[WAV audio]
[Ogg Vorbis]
unsigned)
abalkin commentedon Nov 24, 2008
That's an impressive summary, but what is your conclusion? I don't
see any format that will benefit from a subsecond
timedelta.totimestamp(). Your examples have either multisecond or
submicrosecond resolution.
On Mon, Nov 24, 2008 at 11:00 AM, STINNER Victor <report@bugs.python.org> wrote:
vstinner commentedon Nov 24, 2008
Ooops, timestamp (c) is the *Mac* timestamp: seconds since the 1st
january 1904.
Hum, it's maybe not possible to choose between integer and float. Why
not supporting both? Example:
Attached file (timestamp.py) is a module to import/export timestamp in
all listed timestamp formats. It's written in pure Python.
----------------
It supports int and float types for import and export.
abalkin commentedon Nov 24, 2008
On Mon, Nov 24, 2008 at 12:13 PM, STINNER Victor <report@bugs.python.org> wrote:
..
I would still prefer totimestamp()->(int, int) returning (sec, usec)
tuple. The important benefit is that such totimestamp() will not
loose information and will support more formats than either of your
->int or ->float variants. The ->int can then be spelt simply as
totimestamp()[0] and on systems with numpy (which is likely for users
that deal with floats a lot), totimestamp(microseconds=True) is simply
dot([1, 1e-6], totimestamp()). (and s,us = totimestamp(); return s +
us * 1e-6 is not that hard either.)
vstinner commentedon Nov 24, 2008
Right, I prefer your solution ;-)
abalkin commentedon Nov 24, 2008
On Mon, Nov 24, 2008 at 12:34 PM, STINNER Victor <report@bugs.python.org> wrote:
..
Great! What do you think about extending fromtimestamp(timestamp[,
tz]) and utcfromtimestamp(timestamp) to accept a tuple for the
timestamp?
Also, are you motivated enough to bring this up on python-dev to get a
community and BDFL blessings? I think this has a chance to be
approved.
davidfraser commentedon Nov 24, 2008
----- "STINNER Victor" <report@bugs.python.org> wrote:
There are also the PyWinTime objects returned by PythonWin COM calls which are basically FILETIMEs
I don't have time to get the details now but I recently submitted a patch to make them work with milliseconds - see http://sourceforge.net/tracker/index.php?func=detail&aid=2209864&group_id=78018&atid=551954 (yes I know this is a bit off-topic here)
vstinner commentedon Dec 12, 2008
belopolsky will be happy to see this new version of my patch:
directly the C version of mktime() because time.mktime() creates a
float value
the epoch) to make it consistent with datetime.totimestamp() which
also support this value
(10^-3)
vstinner commentedon Dec 12, 2008
About mktime() -> -1: see the bpo-1726687 (I found the fix in this
issue).
Next job will be to patch datetime.(utc)fromtimestamp() to support
(int, int). I tried to write such patch but it's not easy because
fromtimestamp() will support: int, long, float, (int, int), (int,
long), (long, int) and (long, long). And I don't know if a "long"
value can be converted to "time_t".
AlexanderBelopolsky commentedon Feb 18, 2010
Victor,
As you explain in your own documentation, the proposed method is equivalent to
(time.mktime(self.timetuple()), self.microsecond), so all it does is replacing a less than a one-liner. Moreover, I am not sure time.mktime(self.timetuple()) is something that people would want to do with a TZ-aware datetime. If the tzinfo of the datetime object does not match the system TZ used by mktime, the result will be quite misleading.On the patch itself:
See my comment at bpo-1726687 about the tm_wday == 1 typo.
I don't think time_t to long cast is safe on all platforms.
[-]datetime needs and "epoch" method[/-][+]datetime needs an "epoch" method[/+]mdickinson commentedon Apr 21, 2010
Close bpo-1673409 as a duplicate of this one; combining nosy lists.
vstinner commentedon May 21, 2010
a one-liner, but an horrible one liner :-) I don't like mixing datetime and time modules. I prefer to use only datetime, I prefer its API.
Can you suggest a possible fix to take care of the timezone information? I don't know how to use that.
pitrou commentedon May 21, 2010
I agree with Victor that the APIs need improving, even if it involves providing obvious replacements of obscure one-liners. As an occasional user of datetime and time modules, I have too often wanted to curse those limited, awkwardly inconsistent APIs.
Just my 2 seconds of course :-)
AlexanderBelopolsky commentedon May 21, 2010
On Fri, May 21, 2010 at 7:26 AM, STINNER Victor <report@bugs.python.org> wrote:
..
I believe it should be something like this:
Note the following comment in the documentation for tzinfo.fromutc():
"An example of a time zone the default fromutc() implementation may
not handle correctly in all cases is one where the standard offset
(from UTC) depends on the specific date and time passed, which can
happen for political reasons. The default implementations of
astimezone() and fromutc() may not produce the result you want if the
result is one of the hours straddling the moment the standard offset
changes." I have not tested the above code and it may not work for
non-trivial time-zones.
Still a few questions remain:
should totimestamp() do the same and use given tz for naive datetime
objects?
I believe at this stage we need a python implementation of a prototype
answering these questions and a unit test that would demonstrate how
the prototype would work with nontrivial timezones.
AlexanderBelopolsky commentedon May 21, 2010
On Fri, May 21, 2010 at 7:37 AM, Antoine Pitrou <report@bugs.python.org> wrote:
..
While I agree that the datetime API can be improved, I don't think
Victor's proposal does that. The advantage of an obscure one-liner is
that it is obvious what it does, particularly for someone with a
C/UNIX background. dt.totimestamp() may be easier to write, but it is
entirely non-obvious what it will return. One would expect that
dt.totimestamp() is the inverse of datetime.fromtimestamp(timestamp),
but in timezones with daylight savings adjustments, but such inverse
may not always exist. (01:59AM may be followed by 02:00 AM or by
01:00 AM. so on changeover days datetime(y, m, d, 1, 30).totimestamp()
is either ambiguous or undefined.) As I suggested in my previous
comment, this problem can be resolved, but we are not there yet.
Yes, it would be ideal if a user of datetime module would not need to
reach to other modules for date/time calculations. See also
<http://bugs.python.org/issue6280\>. Do you have other examples of
this sort?
AlexanderBelopolsky commentedon May 21, 2010
On Fri, May 21, 2010 at 11:20 AM, Alexander Belopolsky
<report@bugs.python.org> wrote:
..
s/claendar/calendar/, of course.
pitrou commentedon May 21, 2010
Well, I would argue that the C/Unix legacy in terms of dates and times isn't an example to follow. Python does not force you to use strcat() to concatenate strings, either ;)
But besides, the issue is more how people are supposed to invent that one-liner, let alone remember it easily. Perhaps adding it in the documentation would be a good middle ground, if you think it shouldn't be added to the stdlib.
Well, for example, the datetime module encourages you to use "aware" datetime objects (rather than so-called "naive" objects), but there isn't a single facility to do so. You must reinvent a whole timezone class from scratch.
AlexanderBelopolsky commentedon May 21, 2010
On Fri, May 21, 2010 at 12:20 PM, Antoine Pitrou <report@bugs.python.org> wrote:
..
This is partially addressed by bpo-5094, "datetime lacks concrete
tzinfo impl. for UTC". A more ambitious project would be to add pytz
to stdlib. I believe I've seen this idea discussed and rejected, but
I am not able to find a link to an appropriate thread now. A half-way
project would be to add LocalTimezone given as an example in
http://docs.python.org/dev/py3k/library/datetime.html in addition for
UTC timezone. Any takers?
vivanov commentedon Dec 17, 2010
I'm very disappointed by the outcome of this discussion.
You are committing the biggest sin of modern times - instead of promoting the obtaining and usage of knowledge to solve things, you place restrictions to force the dumbheads into not doing mistakes. The big problem with that is that you can never foresee all usecases and all possible mistakes, thus you will always be sorrily bitten by dumbheads. What does that make of you?
Let me present you a situation - I have a system that passes data via JSON, datetime object is not JSON serializable. For few other reasons, like the epoch and float secs since epoch being defacto standard, and the fact that I absolutely make sure at-the-source that my timestamps are UTC and lack zone awareness, and the fact that I'm not going to display those, but only use them for comparison, and that I'm not going to do historical things and calculations and I don't actually need nanosecond precision, just a tenth of the second, and I'm fine with always using the '<' and '>', not the '==', and the fact that 90% of the cases when use datetimes I have exactly the same requirements and it has always been working fine for me - I choose the lightweight float representation at the one side of the system.
In the SQL DB I use tz unaware timestamps, not floats and my DB access layer returns datetime objects and I prefer them at this end of the system. So I only need to serialize the datetime object. Well, as a matter of fact I have a JSON object serialization already in place for some of my objects, but I do not need that for tz unaware datetimes.
So I look for a method that makes a float from a datetime, which I'm used to in PHP, Java, .NET, C, SQL and you name it. And I'm 2 hours into reading about time, datetime and calendar modules and I still haven't even invented the obscure time.mktime(dt.timetuple())+dt.microseconds*1e-6 . And to even think that this creates a timetuple internally ? I hate it instantly and I dismiss the possibility that the API could be so wrong and I keep searching -> on the internets -> which brings me here where all my illusions are finally buried into the dust.
2 Hours for something, that only needs a few warning lines in the docs?
Ok, the ultimately right thing is to actually serialize the datetime object and rework my other end of the system to use dt instead of float .. maybe .. but not now - now I'm only testing an idea for something completely different and I only need faithful and dutiful Python to give me a float from datetime so I can check something.
I love Python for being simple, logical and consistent and for giving me the tools and not telling me what to do with them.
Not today ... Today Python goes - 'Here is your hammer, but you can not use it to hit straight down. If you hit straight down, and you are using a forge, and you miss your object and hit the forge instead, the hammer could ricochet and hit you back on the forehead, so you can't use it that way. As a matter of fact, there is a gyroscopic sensor embedded in the handle of the hammer and if you try to hit with an angle that is close to 90 degrees, it will detach the head of the hammer from the handle and prevent you from eventually endangering yourself' and I'm like 'WTF??! I'm nailing a nail into a wooden plank!'
Now I'm going to use the obscure one liner and hate it, because it is simply wrong and only someone that doesn't care of implementation detail might think it equal to a proper solution.
The worst thing is, that I learned today, that if I ever need operations with tz aware dates and time intervals in Python, I should actually send an SQL query for that, because my DB has a clean, simple and COMPLETE date/time API that works seamlessly. Yours is a jungle and I see you being asked to include a ready made patch to output a float from a dt, to which you respond by adding a locatime() method 2 years later.
You seriously think, that bpo-9527 solves this? I don't even see a connection.
With bpo-9527 in the python library I would be exactly what I am now - overly frustrated and with the exactly same amount of time completely lost into studying a bunch of tools only to realize that I should avoid using them at all costs.
I'm sorry if I offend somebody by posting this emotional message, I just try to give you another point of view - don't put restrictions and hide the reasoning. Instead, support the thing that is widespread and advise that in certain conditions there are better things to do. And if it doesn't work for some edge cases, or even for half the cases - place a well elaborated warning. Then if programmers still make the mistake - well, let them learn by it. 'Cause that's the way people learn .. they make mistakes. By preventing them from making the mistake, you actually rob them of learning.
bitdancer commentedon Dec 17, 2010
Alexander, I agree with Velko in that it isn't obvious to me how the addition of localtime would answer the desire expressed in this issue. It addresses Antoine's complaint about aware datetimes, but I don't see that it does anything for the "conversion to epoch based timestamp" issue. That is at the very least a documentation issue, since IMO we should be providing our users with the tools they need to interoperate with the systems they need to interoperate with.
Velko: on the other hand, given Victor's research, I don't see float seconds since an epoch appearing anywhere as a standard. Where do you see this being used as a standard? I also don't understand your complaint about the fact that the one-liner creates a timetuple. datetime stores the date and time information as discrete fields, so generating a timetuple is a natural conversion path.
Obviously one could avoid the creation of a Python tuple by calling the C mktime directly in the C code, as has been proposed. I don't see, myself, what would be so bad about providing a 'to_crt_timestamp' method that would, in essence, be the kind of light wrapper around the system API that we provide in so many other places in Python.
pitrou commentedon Dec 17, 2010
Well, given that we already have fromtimestamp(), this sounds like a
poor argument against a totimestamp() method (or whatever it gets
called).
vivanov commentedon Dec 17, 2010
Yes, I didn't mean standard as in RFCed and recommended and dominant, sorry if it sounded that way. I meant just that it is quite common in many places, big and small.
Well, the timetuple is not a tuple, but an object filled with attributes. It contains a few more than are required for this conversion and it doesn't contain one that is required. Therefore I really see that as an inelegant and ineffective way to do the conversion.
AlexanderBelopolsky commentedon Dec 17, 2010
On Fri, Dec 17, 2010 at 9:18 AM, R. David Murray <report@bugs.python.org> wrote:
Conversion of UTC datetime to time stamp is trivial:
There are several reasons not to include this one-liner in stdlib
(other than it being a one-liner).
Different application may need different epoch and retained
precision depends on the choice of the epoch.
The code above works only on naive datetime objects assumed to be
in UTC. Passing say a result of datetime.now() to it is likely to
result in a hard to find bug.
While it is not hard to extend the timestamp(t) code to cover aware
datetime objects that use fixed offset tzinfo such as those with
tzinfo set to a datetime.timezone instance, it is not well defined for
the "smart" tzinfo implementations that do automatic DST adjustment.
This is where the localtime (bpo-9527) issue comes into play.
pitrou commentedon Dec 17, 2010
But then why does fromtimestamp() exist?
And returning a (seconds, microseconds) tuple does retain the precision.
So, if the "trivial" code doesn't work, you can't bring it up as an
argument against shipping this functionality, right?
Still, fromtimestamp() exists and apparently fulfills people's
expectations. So why can't the same strategy be used for totimestamp()
as well?
AlexanderBelopolsky commentedon Dec 17, 2010
On Fri, Dec 17, 2010 at 12:17 PM, Antoine Pitrou <report@bugs.python.org> wrote:
..
A better question is why datetime.utcfromtimestamp(s) exists given
that it is actually longer than equivalent EPOCH + timedelta(0, s)? I
am not responsible for either of these methods, but at least
datetime.fromtimestamp(s, tz) is well defined for any timezone and
timestamp unlike its inverse.
It does, but it does not help much those who want a float - they would
still need another line of code. Note that with divmod(timedelta,
timedelta), you can now easily extract (seconds, microseconds) or
any other tuple like (weeks, days, seconds. microseconds) from
timedelta objects. See msg75904 above.
Well, no one has come up with the code that does work so far. Note
that timetuple path does not work either because it does not fill
tm_isdst correctly. The only solution I can think of for having
proper inverse to fromtimestamp() is to add isdst to datetime objects.
This would allow correct round-tripping between datetime and
timetuple and datetime and timestamp.
Because in certain timezones fromtimestamp() can return the same
datetime value for different timestamps and some datetime values do
not have a corresponding timestamp. I have not seen a working
proposal on how to handle these issues yet. You are asking to provide
an inverse to an existing function simply because the function exists.
But the function in question is not invertible.
pitrou commentedon Dec 17, 2010
??? EPOCH is not even a constant in the datetime module.
And regardless, the point is *not* the number of characters typed, but
how easy it is to come up with the solution. Calling the appropriate
(and appropriately-named) method is much easier than coming up with the
right datetime arithmetic incantation. It's Python, not Perl. "There
should be one obvious way to do it".
Yes, but a very obvious one at least.
Do you think many users even think of calling divmod() timedelta
objects? I don't, personally.
You apparently hold the opinion that the datetime module should be
reserved for experts in arithmetic over dates, times and timedeltas. But
it's not. It's the Python stdlib and it should provide reasonably
high-level tools to do the job.
AlexanderBelopolsky commentedon Dec 17, 2010
On Fri, Dec 17, 2010 at 1:17 PM, Antoine Pitrou <report@bugs.python.org> wrote:
..
No, and it does not belong there. A higher level library that uses
seconds since epoch for interchange may define it (and make a decision
whether it should be a naive datetime(1970, 1, 1) or datetime(1970, 1,
1, tzinfo=timezone.utc)).
I don't see anything obvious about the choice between
utcfromtimestamp(s), fromtimestamp(s) and utcfromtimestamp(s,
timezone.utc).
datetime(1970, 1, 1) + timedelta(seconds=s)
is obvious, self-contained, short and does not require any knowledge
other than elementary school arithmetic to understand. Compared to
this, "utcfromtimestamp" is a monstrosity that suggests that something
non-trivial, such as UTC leap seconds is been taken care of.
Let's see:
and
which one is *obviously* correct? Are they *obviously* equivalent?
Note that when timedelta.total_seconds() was first committed, it
contained a numerical bug. See bpo-8644.
Sure, but if the goal is to implement json serialization of datetime
objects, maybe stdlib should provide a high-level tool for *that* job?
Using float representation of datetime is probably the worst option
for json: it is non-standard, may either loose information or
introduce spurious differences, and is not human-readable.
In any case, you ignore the hard question about totimestamp():
fromtimestamp() is not invertible in most real life timezones. If you
have a solution that does not restrict totimestamp() to UTC, I would
like to hear it. Otherwise, I don't see any problem with (t -
datetime(1970, 1, 1)).total_seconds() expression. Maybe we can add
this recipe to utcfromtimestamp() documentation.
pitrou commentedon Dec 17, 2010
And so what was your point exactly?
I don't think the "time" module can be named "higher level", and it
still handles such timestamps.
Sigh.
Again: it's so obvious that you're the only one who seems to easily come
up with those solutions. How many times does it have to be repeated?
I don't see anything suggesting it is a monstrosity. The name is
grammatically bizarre, but that's all.
Both are obviously correct for whatever the non-perverted user has in
mind. People in real life don't care whether they will retain
microsecond precision when carrying a floating point timestamp around.
For the simple reason that the data source itself will not have such
precision.
So? What is your point?
IMO, the solution would have the datetime object carry the offset from
UTC with it, rather than try to be smart and compute it dynamically.
AlexanderBelopolsky commentedon Dec 17, 2010
On Fri, Dec 17, 2010 at 2:35 PM, Antoine Pitrou <report@bugs.python.org> wrote:
..
Remember, most of the code is written once, but read and edited many
times. Show me one person who will have trouble understanding what
datetime(1970, 1, 1) + timedelta(seconds=s) means and show me another
who can understand datetime.utcfromtimestamp(s) without reading the
manual.
Yes, UTC not being a proper acronym in any human language is one
problem, Python datetime not being able to represent some valid UTC
times is another.
That's correct, but most users expect their timestamps to be the same
when saved on one system and read on another. Granted, most users
expect the same from their floats as well, but this can only be solved
by education. Calendaric calculations are complex enough that we
don't want to expose users to floating point gotchas at the same time.
I thought the point was obvious: conversion between time values and
float is non-trivial and error prone. Users should not be encouraged
to casually convert (seconds, microseconds) tuples to floats. If they
do, chances are they will do it differently in different parts of the
program.
Ditto. This is exactly what bpo-9527 is attempting to achieve.
pitrou commentedon Dec 17, 2010
Ok. Too bad you don't live on the same planet than most of us. I bail
out.
abalkin commentedon Dec 17, 2010
On Fri, Dec 17, 2010 at 3:26 PM, Antoine Pitrou <report@bugs.python.org> wrote:
..
Sorry that my attempt at humor has proven to be too subtle. I was
referring to the following fact:
"""
The International Telecommunication Union wanted Coordinated Universal
Time to have the same symbol in all languages. English and French
speakers wanted the initials of both their respective language's terms
to be used internationally: "CUT" for "coordinated universal time" and
"TUC" for "temps universel coordonné". This resulted in the final
compromise of "UTC".
"""
http://en.wikipedia.org/wiki/Coordinated_Universal_Time
vstinner commentedon Dec 17, 2010
It looks like it's not possible to choose between float and (int, int) output type for datetime.totimestamp(). One is more practical (and enough for people who doesn't need an exact result), and one is needed to keep the same resolution than the datetime object. I think that we can add two methods:
I choosed the shortest name for float because I suppose that most users prefer float than a tuple, and so the API is symmetrical:
My patch have to be updated to use the timezone (and the DST thing?) and also to update the Python implementation.
ping commentedon Mar 31, 2011
I am extremely disappointed by what has happened here.
We are talking about a very simple method that everybody needs, and that has been reimplemented over and over again. I have been frustrated countless times by the lack of a utctotimestamp() method. I have watched beginners and experienced programmers alike suffer over and over again for the lack of this method, and spend hours trying to figure out why Python doesn't have it and how it should be spelled in Python.
The discussion here has been stuck on assumptions that the method must meet all of the following ideals:
These ideals cannot all be met simultaneously and perfectly. The correct thing to do as an engineer is to choose a practical compromise and document the decision.
The compromise that almost everyone chooses (because it is useful, convenient, has microsecond precision at least until the year 2100, and millisecond precision is frequently sufficient) is to use a floating-point number with an epoch of 1970-01-01. Floating-point seconds can be easily subtracted, added, serialized, and deserialized, and are a primitive data type in nearly every language and database. They are unmatched in ease of use. So everyone wastes time searching for the answer and figuring out how to write:
We should use this as the definition of datetime.utctotimestamp(), document its limitations, and be done with it.
Instead, this essential and useful method has now been held up for almost three YEARS by an inability to accept a simple engineering decision. Unbelievable.
abalkin commentedon Mar 31, 2011
On Thu, Mar 31, 2011 at 2:52 PM, Ka-Ping Yee <report@bugs.python.org> wrote:
..
What exactly are you disappointed about? As far as I can tell, the
feature request has not been rejected, just no one has come up with a
satisfactory solution. The issue is open and patches are welcome.
This is not what this issue has been about so far. It was about local
time to timestamp. In py3k, utctotimestamp() is easy:
These "beginners and experienced programmers" may want to reconsider
using floating point numbers to store high precision timestamps. I
know that some OSes made the same unfortunate choice in system APIs,
but it does not make this choice any better. I can make a long list
of why this is a bad choice, but I'll just mention that the precision
of your timestamp varies from year to year and the program that works
fine today may mysteriously fail in 5 years when nobody is around who
can fix it anymore.
No it was actually stuck because of the inability to reliably obtain
the system UTC offset for historical times. This is a solvable
problem, but the patches proposed so far did not solve it correctly.
On top of this, there is an issue of datetime.fromtimestamp() not
being invertible in the presence of DST shifts, so
datetime.totimestamp() is ambiguous for some datetime values.
Those who need to do arithmetics on time values more often deal with
durations rather than points in time. An arbitrary epoch around
current time is often more appropriate for timeseries analytics than
Unix epoch.
Compared to what? I find integers much more suitable for representing
points in time than floats. Yes, in some languages you have to deal
with 32-bit int overflow issues if you want to be able to deal with
durations of over 100 years expressed in microseconds, but these days
64-bit integers are almost universally available.
And this is the wrong answer. Someone else using (dt -
EPOCH).total_seconds() may get a slightly different result. Some may
argue that given that it is not obvious what expression to use, we
need to provide a function. However, we already provided
timedelta.total_seconds() that hides the floating point details. In
my opinion, even adding total_seconds() was a mistake and x /
timedelta(seconds=1) is just as short and more explicit than
x.total_seconds().
I think the best we can do is to expand datetime.utcfromtimestamp()
documentation to explain that it is equivalent to
and either leave it as an exercise to the reader to solve
utcfromtimestamp(s) = dt for s or spell out
ping commentedon Apr 2, 2011
Plenty have proposed a satisfactory solution. No one has come up with a solution that is satisfactory to *you*, because you have overconstrained the problem. The reason we still have no utctotimestamp() after all these years is that you, and you alone as far as I know, refuse to accept a method that inverts utcfromtimestamp() with microsecond precision over its working range. Such a method is a perfectly reasonable and acceptable solution and would add a lot of value to Python as a language.
I suspect you don't realize just how much pain you have unintentionally caused the world of Python users by singlehandedly blocking progress on this issue. I've seen them: students, friends, coworkers -- even very smart and capable people are stymied by it. No one thinks of looking in the calendar module. Maybe if you watched some of them struggle with this, you would understand.
To take this perspective is to miss the point of Python.
JayTaylor commentedon Apr 4, 2011
I couldn't agree more with ping's position on this. It is against the spirit of what Python has set out to be, and the blocking needs to stop.
Any chance we could get a .epoch() function into python 2.7 as well?
abalkin commentedon Apr 5, 2011
On Mon, Apr 4, 2011 at 5:42 PM, Jay Taylor <report@bugs.python.org> wrote:
..
Adding votes to a tracker issue without a working patch will not move
it any further. There are several committers besides me in the nosy
list including the original author of the datetime module. If it was
such a universally desired feature as Ka-Ping makes it sound, it would
be committed long before I became the maintainer of the datetime
module.
I don't think any committer has a power to *block* a patch. I
certainly don't. If Ka-Ping wants to add a feature over my
objections, it is well within his power to do so. (Note that I
objected to timedelta.total_seconds(), but it was added nevertheless.)
It would be best, however to bring this to python-dev or python-ideas
first.
No.
malemburg commentedon Apr 5, 2011
Just to add another data point to this discussion:
mxDateTime, which in large parts inspired the Python datetime module,
has had a .ticks() method (for local time) and a .gmticks() method
(for UTC) for more than a decade now and so far, I haven't seen a
single complaint about any of the issues raised in this discussion.
The methods naturally return the Unix ticks value as float,
since that's what the time module uses as basis and the whole purpose
of those methods is to make interaction with the time module easy
and straight-forward. Likewise, the epoch is also the same as the time
module's one.
Victor's patch could easily be updated to return floats as well,
to make it compatible with the time module.
There's only one catch that Victor's patch doesn't include: mktime()
doesn't always work with DST set to anything but -1. mxDateTime
checks the API at module load time and then determines whether
it can be used with a DST setting or not (see the mxDateTime code
for details). Not sure whether today's mktime() implementations
still have any issues with this, but it's better to double-check
than to get wrong results.
vstinner commentedon Apr 5, 2011
Marc, could you maybe write a new patching taking care of the DST and
maybe also the timezone? It looks like you have a long experience in
timestamps :-)
malemburg commentedon Apr 5, 2011
STINNER Victor wrote:
Sorry, but no. I'm not really a fan of the datetime module and
try to stay away from it whenever I can :-)
Note that dealing with DST in timezones other than the local time
zone, is bound to go wrong without direct access to the tz library.
The C lib doesn't provide any good way to access timezone
information other than the local timezone or UTC.
When dealing with date/time values, it is usually best to stay with
UTC and only transform those values into local times in user
interfaces on the front-end client.
Consequently, I'd suggest to only allow UTC and local timezone
conversions for the method in the datetime module.
abalkin commentedon Apr 5, 2011
On Tue, Apr 5, 2011 at 4:33 AM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
..
Yes, mxDateTime's gmticks()/ticks() pair of functions present a much
more mature design than anything that has been proposed here. It is
telling, however, that no one has mentioned mxDateTime's gmticks() on
this issue in four years. On a duplicate bpo-1673409, Marc did
bring it up, but as far as I can tell, no one responded. See
msg75411.
Google code search,
http://www.google.com/codesearch?hl=en&sa=N&q=gmticks+lang:python
returns only 13 hits for "gmticks". In several instances, the
resulting float is immediately converted to int, in other instances
"gmticks" is mentioned in comments and the code works around its bugs.
I would not use Google Code search as an ultimate arbiter on the
popularity of a feature, so I would really like to hear from the
proponents about real life uses of gmticks() or any other examples
where a similar method "has been reimplemented over and over again."
Well, search for gmticks does not return too many hits outside of
mxDateTime code and manuals, but I had no trouble finding this:
"""
okay, all the MySQLdb dataobject tick-based time handling methods are
broken in various ways -- reconstruct GMT ticks from time module's
mktime...
"""
http://viewvc.tigris.org/ds/viewMessage.do?dsForumId=4251&dsMessageId=656863
Follow the link for some more colorful language describing developer's
experience with the feature.
Note that it is likely that the bug MySQLdb developer complained about
was fixed in mxDateTime at some point,
<http://www.egenix.com/www2002/python/mxDateTime-History.html\>, but
this shows that implementing gmticks() correctly is not as trivial as
those who never tried might think.
Which in turn is a mistake IMO. Note that POSIX does not use float
timestamps for a reason.
This is not the goal that I would support. I would rather see code
that uses datetime module not require time module methods at all.
Victor reported implementing two methods, one to return a float and
another to return a tuple. See msg124259. I am not sure I've seen
that code.
No, it is not about "only one catch". Victor's patch is simply wrong.
For an aware datetime instance it extracts DST flag from tzinfo, but
ignores the offset.
abalkin commentedon Apr 5, 2011
MAL> Since most of the datetime module was inspired by mxDateTime,
MAL> I wonder why [ticks()/gmticks()] were left out. (msg75411)
"""
The datetime module intended to be an island of relative sanity.
Because the range of dates "timestamps" can represent varies across
platforms (and even "the epoch" varies), datetime doesn't even try to
produce timestamps directly -- datetime is more of an alternative to
"seconds from the epoch" schemes. Because datetime objects have
greater range and precision than timestamps, conversion is
problem-free in only one direction. It's not a coincidence that
that's the only direction datetime supplies ;-)
""" - Tim Peters
http://bytes.com/topic/python/answers/522572-datetime-timestamp
I will also add that fromtimestamp() is not invertible in the presence of DST. That's why mxDatetime.ticks() takes a DST flag making it effectively a multi-valued function. Note that naive users, who just want to pass datetime values to an OS function expecting a float, most likely will not have means of properly obtaining DST flag.
malemburg commentedon Apr 5, 2011
Alexander Belopolsky wrote:
mxDateTime needs those two methods, since it doesn't natively
use timezones. The .ticks() method is used for local time values,
.gmticks() for UTC ones; that's why there are two methods.
The .gmticks() method is always used when storing UTC values
in mxDateTime instances, which actually is the preferred way
of storing data in databases. Google Code doesn't really count
much, since it only scans a limited number of OSS code bases.
Most of our users are commercial users who use the tools
in-house.
Note that implementing .gmticks() is fairly easy on POSIX
conform systems. On most others, timegm() can be used. If
that doesn't exist, things get tricky, but that case should
be rare nowadays.
Note that he was referring to the .ticks() method, not the .gmticks()
method. The patch doesn't say which version of mxDateTime he was
using. The bug mentioned in the changelog was fixed in 1998. It is
possible, however, that the mktime() on his system was broken - which
is why I added a test for it in mxDateTime.
The time module is our reference in this case and this tries hard
to add fractions of a second to the value :-)
Note that sub-second accuracy relies on a number of factors,
the storage format most certainly is the least important
aspect ;-)
On many systems, you only get 1/100s accuracy, on others, the timer
ticks in fixed increments, giving you even weirder sub-second values
(e.g. time appears to stay constant between time.time() calls).
OTOH, there's a new set of APIs for nano-second accuracy available
now, which the datetime module objects cannot represent at all due
to the integer-based storage format.
BTW: The integer format was chose in order to keep the memory
footprint of the objects low.
No chance :-) In practice, the time module gets used a lot
for date/time storage or to quickly measure time deltas.
Some people also prefer time module ticks due to their lower
memory footprint, esp. when it comes to storing thousands of
time values in time series.
I had a look at the last patch on this ticket.
True, so make that two issues ;-)
malemburg commentedon Apr 5, 2011
Alexander Belopolsky wrote:
IMHO, the whole concept of DST is broken, but that's not our fault :-)
Ditching the concept just because it is known to fail for
one hour out of 8760 you have in a typical year doesn't really
warrant breaking the "practicality beats purity" guideline.
Otherwise, we'd have to ditch the date support in the datetime module
too: after all, Feb 29 only exists every 4 years (well, most of the
time) - and that's one day out of 1461 in those 4 years, so an
even worse ratio :-)
And I'm not even starting to talk about ditching the concept
of Unix ticks to begin with, as a result of having leap seconds
causing POSIX ticks values not matching (real) UTC ticks.
In reality, all these things hardly ever matter and if they do,
users will either know that they have to make conscious decision,
simply don't care or decide not to care.
BTW: A "timestamp" usually refers to the combination of date and
time. The time.time() return value is "seconds since the Epoch".
I usually call those values "ticks" (not sure whether it's standard
term of not, but always writing "seconds since Epoch" wasn't an
option either ;-)).
Date/time is fun, isn't it ?
abalkin commentedon Apr 5, 2011
Let me state my position on this issue once again. Converting datetime values to float is easy. If your dt is a naive instance representing UTC time:
If your dt is an aware instance:
These recipes are easy to adjust for your application needs. One application may want millisecond or microsecond ticks, another might want to carry subsecond presision in a separate integer, third may want to avoid timestamps before 1970 or after 2038 or ignore microseconds altogether. No matter what a hypothetical datetime.epoch() will provide, most of applications will need to add a line or two to its code to serve their needs. Applications that will use dt.epoch() naively without thinking what dt represents (say local or UTC) will be buggy.
The only related feature that I think is missing from datetime module is the ability to obtain local time as an aware datetime instance and to convert a naive datetime instance assumed to represent local time to an aware one.
This is the subject of bpo-9527, but there is a resistance to adding that feature.
abalkin commentedon Apr 5, 2011
On Tue, Apr 5, 2011 at 1:45 PM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
..
In Unix context, the term "timestamp" is usually associated with the
various time values that OS stores with the files. I think this use
is due to the analogy with physical "received on" timestamps used on
paper documents. Since it is well-known that Unix filesystems store
time values as seconds since Epoch, it is common to refer to these
values as "Unix timestamps".
See, for example:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/touch.html
vivanov commentedon Apr 7, 2011
On 04/05/2011 18:22, Alexander Belopolsky wrote:
Refusing to cooperate with the rest of the world is not sane by my books.
On 04/05/2011 21:06, Alexander Belopolsky wrote:
Please add these lines to the datetime module's documentation. In some
central, well lit place. I believe that if nothing else, the whole
discussion should have proved to you that there are many people looking
for them.
OTOH a sinceepoch(epoch=datetime(1970,1,1)) method of the datetime class
should be equally easy. Would be especially useful if few of the more
frequently used EPOCHs are provided as constants.
abalkin commentedon Apr 7, 2011
On Thu, Apr 7, 2011 at 6:20 AM, Velko Ivanov <report@bugs.python.org> wrote:
..
This is precisely what I suggested at the end of msg132697 above. See
attached patch (bpo-2736-doc.diff) for a proposed documentation
enhancement.
python-dev commentedon Apr 25, 2011
New changeset b55eac85e39c by Alexander Belopolsky in branch 'default':
Issue bpo-2736: Documented how to compute seconds since epoch.
http://hg.python.org/cpython/rev/b55eac85e39c
python-dev commentedon Jun 8, 2012
New changeset 6671c5039e15 by Alexander Belopolsky in branch 'default':
Issue bpo-2736: Added datetime.timestamp() method.
http://hg.python.org/cpython/rev/6671c5039e15