Add aware local time support to datetime module

@tim-one

BPO	9527
Nosy	@tim-one, @loewis, @warsaw, @jribbens, @rhettinger, @jcea, @amauryfa, @mdickinson, @ncoghlan, @abalkin, @pitrou, @catlee, @vstinner, @cameron-simpson, @merwok, @bitdancer
Dependencies	bpo-1647654: No obvious and correct way to get the time zone offset bpo-1667546: Time zone-capable variant of time.localtime
Files	datetime-localtime-proto.diff datetime-localtime-proto-1.diff datetime-astimezone-proto.diff testtz.py: A simple test in a non-trivial timezone issue9527.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = <Date 2012-06-22.17:37:39.663>
created_at = <Date 2010-08-06.03:48:18.582>
labels = ['extension-modules', 'type-feature', 'library']
title = 'Add aware local time support to datetime module'
updated_at = <Date 2012-06-22.20:09:58.207>
user = 'https://github.com/abalkin'

bugs.python.org fields:

activity = <Date 2012-06-22.20:09:58.207>
actor = 'python-dev'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2012-06-22.17:37:39.663>
closer = 'belopolsky'
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2010-08-06.03:48:18.582>
creator = 'belopolsky'
dependencies = ['1647654', '1667546']
files = ['18410', '20410', '25939', '25940', '26062']
hgrepos = []
issue_num = 9527
keywords = ['patch']
message_count = 21.0
messages = ['113067', '113069', '118675', '126292', '126295', '127516', '127517', '135227', '153928', '161643', '162631', '162658', '162660', '163136', '163137', '163297', '163429', '163442', '163465', '163467', '163476']
nosy_count = 29.0
nosy_names = ['tim.peters', 'loewis', 'barry', 'jribbens', 'rhettinger', 'jcea', 'pboddie', 'jamesh', 'guettli', 'amaury.forgeotdarc', 'mark.dickinson', 'ncoghlan', 'davidfraser', 'belopolsky', 'pitrou', 'andersjm', 'catlee', 'vstinner', 'techtonik', 'tomster', 'werneck', 'hodgestar', 'Neil Muller', 'cameron', 'eric.araujo', 'erik.stephens', 'steve.roberts', 'r.david.murray', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue9527'
versions = ['Python 3.2']

See python-dev post for motivation.

http://mail.python.org/pipermail/python-dev/2010-August/102842.html

I am attaching a patch implementing the proposed method in datetime.py. I will also paste the code below. Note that this is only prototype. Real implementation will use tm_zone and tm_gmtoff components of tm structure on platforms that supply them.

    @classmethod
    def localtime(cls, t=None, isdst=-1):
        """Return local time as an aware datetime object.                                                                                                          
                                                                                                                                                                   
        If called without arguments, return current time.  Otherwise                                                                                               
        *t* is converted to local time zone according to the system                                                                                                
        time zone database.  If *t* is naive (i.e. t.tzinfo is None),                                                                                              
        it is assumed to be in local time.  In this case, a positive or                                                                                            
        zero value for *isdst* causes localtime to presume initially                                                                                               
        that summer time (for example, Daylight Saving Time) is or is                                                                                              
        not (respectively) in effect for the specified time.  A                                                                                                    
        negative value for *isdst* causes the localtime() function to                                                                                              
        attempt to divine whether summer time is in effect for the                                                                                                 
        specified time.                                                                                                                                            
        """
        if t is None:
            t = _time.time()
        else:
            if t.tzinfo is None:
                tm = t.timetuple()[:-1] + (isdst,)
                t = _time.mktime(tm)
            else:
                delta = t - datetime(1970, 1, 1, tzinfo=timezone.utc)
                t = delta.total_seconds()
        tm = _time.localtime(t)
        if _time.daylight:
            if tm.tm_isdst:
                offset = _time.altzone
                tzname = _time.tzname[1]
            else:
                offset = _time.timezone
                tzname = _time.tzname[0]
        tz = timezone(timedelta(seconds=-offset), tzname)
        return cls.fromtimestamp(t, tz)

Merging nosy lists from issues bpo-1647654 and bpo-2736. If datetime.localtime() is implemented, I argue that the features requested in these two issues will become unnecessary.

Would anyone like to review this? The Rietveld link works (thanks Martin!) and I would like to get some feedback on the python version before I invest effort into coding this in C.

Following Anatoly's review, I renamed datetime argument and a local variable, added comments and expanded docstring. I am uploading a new patch: datetime-localtime-proto-1.diff.

Martin, I could not figure out how to add the new patch to rietveld and I don't think auto-review feature works.

Forwarding Rietveld conversation to the tracker. It looks like
Rietveld integration has a bug and sends reviews to
None@psf.upfronthosting.co.za rather than to report@bugs.python.org.

Forwarded conversation
Subject: Add aware local time support to datetime module (bpo-9527)
------------------------

From: <techtonik@gmail.com>
Date: Thu, Jan 13, 2011 at 7:14 PM
To: belopolsky@users.sourceforge.net
Cc: None@psf.upfronthosting.co.za

I'm not proficient in C and not an expert in datetime issues either, but
because nobody else stepped in to do review, I've left some comments.
Perhaps making the issue more acceptable by general public will help to
close it faster.

http://bugs.python.org/review/9527/diff/1568/2704
File Lib/datetime.py (right):

http://bugs.python.org/review/9527/diff/1568/2704#newcode1420
Lib/datetime.py:1420: def localtime(cls, t=None, isdst=-1):
The t param should probably be called 'time'.

http://bugs.python.org/review/9527/diff/1568/2704#newcode1421
Lib/datetime.py:1421: """Return local time as an aware datetime object.
ISTM that we need an object hierarchy like DateTime->DateTimeTZ, because
debugging which object is TZ aware and which is not is rather hard.

http://bugs.python.org/review/9527/diff/1568/2704#newcode1435
Lib/datetime.py:1435: t = _time.time()
Here we should always receive aware object, but I couldn't find
reference to C standard to ensure that all systems return this
correctly.

http://bugs.python.org/review/9527/diff/1568/2704#newcode1437
Lib/datetime.py:1437: if t.tzinfo is None:
I'd replace this with elif and place comment that here we detect
non-aware time object. Docstring above is nice, but it will probably be
parsed into documentation, and I am not sure if these details should be
present in user manual.

http://bugs.python.org/review/9527/diff/1568/2704#newcode1438
Lib/datetime.py:1438: tm = t.timetuple()[:-1] + (isdst,)
IIUC return result of time() value is time.struct_time which doesn't
have timetuple() method. If you work with datetime objects only, then
you need to name variables accordingly.

http://bugs.python.org/review/9527/show

----------
From: <techtonik@gmail.com>
Date: Fri, Jan 14, 2011 at 12:28 PM
To: belopolsky@users.sourceforge.net
Cc: None@psf.upfronthosting.co.za

On 2011/01/14 04:30:11, sasha wrote:

Can you comment on whether you find the
proposed function useful? Is the interface intuitive?

I prefer to threat UTC time and timezone separately. My API wannabe:

global.time() - returns value that can be printed and it will be in UTC
global.offset() - difference in seconds between current timezone and UTC
(so that +2 will be 7200 seconds and -2 is -7200)
global.fromiso() - returns value parsed from ISO 8601 format, probably
pair (UTC time, offset)
global.toiso(time, [offset]) - for symmetry

Maybe its even better if global.time() returns tuple (UTC time, offset)

As you may see I am not interested in DST details etc. All I need is the
ability to parse and generate timestamps. If your datetime.localtime()
returns datetime object, how should I use it to generate Atom timestamp
with proper TZ modifier?

http://www.atomenabled.org/developers/syndication/atom-format-spec.php#date.constructs

In my low-level and non-OOP API it is:

global.toiso(global.time(), global.offset())

epoch as an

argument (unlike the eponymous time module function). It takes an
arbitrary
datetime instance and converts it to an aware datetime instance with
tzinfo set
to an appropriate fixed offset timezone.

At first I thought about datetime.toaware(dt), but this description of
yours is better than the one provided in docstring. If
datetime.localtime([dt]) gets local time in timezone aware object or
converts existing datetime (how about other types?), then the name
datetime.localtime() seems fine. But I would consider alternative
datetime.globaltime() name with the meaning that we get datetime object
that is globally synchronized, in other words it can be used not only
locally. .localtime() is ambiguous in this respect.
On 2011/01/14 04:30:11, sasha wrote:

implementation. I would not call it "time" because that conflicts
with the
time class and localtime expects a datetime instance in t.

dt is a consistent name for datetime parameters in Python manual.
On 2011/01/14 04:30:11, sasha wrote:

Early versions of datetime has a separate datetimetz class. I don't
remember
what were the actual reasons for removing it, but I am glad that this
was done.
Hopefully applications will stop using naive datetime objects at some
point, so
there will be less need to distinguish naive and aware instances.

It must be recorded for history if we want to get rid of date/time curse
in Python4 without missing any details. Removing naive objects in favor
of normalized objects with UTC timestamps seems like a way to go.
On 2011/01/14 04:30:11, sasha wrote:

No. _time.time() returns seconds since epoch. Naive/aware distinction
is not
applicable.

But these seconds are in UTC. There is no TZ correction.

I think Tim and Guido had deliberately avoided local timezone awareness when they designed the module.

Also, I question whether the proposed API is correct. ISTM that any code that sets the *dst* parameter is guaranteed to be wrong (hardwiring-in a value that will change every few months).

Have you studied the solutions used in other libraries and other languages? This path has been well-traveled, so original work may be less helpful than just researching existing solutions.

On Sat, Jan 29, 2011 at 11:56 PM, Raymond Hettinger
<report@bugs.python.org> wrote:
..

Also, I question whether the proposed API is correct. ISTM that any code that sets the *dst*
parameter is guaranteed to be wrong (hardwiring-in a value that will change every few months).

Can you elaborate on this? ISTM that your argument would equally
apply to C/POSIX mktime() API. It won't be the first time C/POSIX got
time handling wrong, but I doubt it is the case in this instance.

LocalTimezone support would be *really* helpful for the email module. It would allow us to have unambiguous semantics for datetime objects representing timestamps exacted from or inserted into email messages (see bpo-665194 for recent discussion). The email module is already trying to handle timestamp translation, and I'd be willing to bet it is buggier than the proposal here.

At one point I even started to copy the LocalTimezone class from the docs into the email module. I implemented a naive extension of the current formatdate function instead, but after Alexander's feedback on bpo-665194 I think the naive implementation is not a good idea.

One important thing to remember is that once a time is in the past, whether or not DST was in effect for that time *is never going to change*. That means converting a DST aware timezone to a fixed offset timezone will work just fine for historical times.

It's mainly applications that need to handle times in the *future* (where the DST dates are still subject to change) that have to be aware of the problems when trying to handle DST with fixed offset timezone objects.

I think it's sufficient if the documentation pushes developers of such applications towards modules like pytz and dateutil to get access to variable offset tzinfo implementations.

New changeset df12ce0c96eb by R David Murray in branch 'default':
bpo-665194: Add a localtime function to email.utils.
http://hg.python.org/cpython/rev/df12ce0c96eb

This was originally posted on python-dev, but I hope reposting it here will make this issue easier to navigate.

"""
With addition of fixed offset timezone class and the timezone.utc
instance [0], it is easy to get UTC time as an aware datetime
instance:

>>> datetime.now(timezone.utc)
datetime.datetime(2010, 8, 3, 14, 16, 10, 670308, tzinfo=datetime.timezone.utc)

However, if you want to keep time in your local timezone, getting an
aware datetime is almost a catch 22. If you know your timezone UTC
offset, you can do

>>> EDT = timezone(timedelta(hours=-4))
>>> datetime.now(EDT)
datetime.datetime(2010, 8, 3, 10, 20, 23, 769537,
tzinfo=datetime.timezone(datetime.timedelta(-1, 72000)))

but the problem is that there is no obvious or even correct way to
find local timezone UTC offset. [1]

In a comment on issue bpo-5094 ("datetime lacks concrete tzinfo
implementation for UTC"), I proposed to address this problem in a
localtime([t]) function that would return current time (or time
corresponding to the optional datetime argument) as an aware datetime
object carrying local timezone information in a tzinfo set to an
appropriate timezone instance. This solution is attractive by its
simplicity, but there are several problems:

An aware datetime cannot carry all information that system
localtime() supplies in a time tuple. Specifically, the is_dst flag
is lost. This is not a problem for most applications as long as
timezone UTC offset and timezone name are available, but may be an
issue when interoperability with the time module is required.
Datetime's tzinfo interface was designed with the idea that
<2010-11-06 12:00 EDT> + <1 day> = <2010-11-07 12:00 EST>, not
<2010-11-07 12:00 EDT>. It other words, if I have lunch with someone
at noon (12:00 EDT) on Saturday the day before first Sunday in
November, and want to meet again "at the same time tomorrow", I mean
12:00 EST, not 24 hours later. With localtime() returning datetime
with tzinfo set to fixed offset timezone, however, localtime() +
timedelta(1) will mean exactly 24 hours later and the result will be
expressed in an unusual for the given location timezone.

An alternative approach is the one recommended in the python manual.
[3] One could implement a LocalTimezone class with utcoffset(),
tzname() and dst() extracting information from system mktime and
localtime calls. This approach has its own shortcomings:

While adding integral number of days to datetimes in business
setting, it is natural to expect automatic timezone adjustments, it is
not as clearcut when adding hours or minutes.
The tzinfo.utcoffset() interface that expects *standard* local time
as an argument is confusing to many users. Even the "official"
example in the python manual gets it wrong. [4]
datetime(..., tzinfo=LocalTimezone()) is ambiguous during the
"repeated hour" when local clock is set back in DST to standard time
transition.

As far as I can tell, the only way to resolve the last problem is to
add is_dst flag to the datetime object, which would also be the
only way to achieve full interoperability between datetime objects and
time tuples. [5]

The traditional answer to a call for improvement of timezone support in
datetime module has been: "this is up to 3rd parties to implement."
Unfortunately, stdlib is asking 3rd parties to implement an impossible
interface without giving access to the necessary data. The
impossibility comes from the requirement that dst() method should find
out whether local time represents DST or standard time while there is
an hour each year when the same local time can be either. The missing
data is the system UTC offset when it changes historically. The time
module only gives access to the current UTC offset.

My preference is to implement the first alternative - localtime([t])
returning aware datetime with fixed offset timezone. This will solve
the problem of python's lack of access to the universally available
system facilities that are necessary to implement any kind of aware
local time support.

[0] http://docs.python.org/dev/library/datetime.html#timezone-objects
[1] http://bugs.python.org/issue1647654
[2] http://bugs.python.org/issue5094#msg106997
[3] http://docs.python.org/library/datetime.html#tzinfo-objects
[4] http://bugs.python.org/issue9063
[5] http://bugs.python.org/issue9004
"""

-- http://mail.python.org/pipermail/python-dev/2010-August/102842.html

Two objections have been raised to the proposed datetime.localtime() function:

It offers the third subtly different way to obtain current time in datetime module. The first two being provided by datetime.now() and datetime.today().
C library localtime function takes POSIX timestamp as an argument, so datetime.localtime() should follow suit.

I attach a prototype patch for a different approach: make datetime.astimezone() method supply local timezone information if no argument is given.

This patch also demonstrates that extracting all TZ information that platform may have knowledge of is not trivial.

One problem I can see with using a fixed offset tzinfo for localtime is that it might confuse people when doing date arithmetic. For example:

>>> d = datetime.localtime() + timedelta(days=7)

While it will give a correct answer as a point in time it will have the wrong time zone offset if run just before a daylight saving transition, which could be just as confusing.

I'm not sure how you'd solve this without e.g. importing pytz into the standard library though.

On Mon, Jun 11, 2012 at 11:18 PM, James Henstridge <report@bugs.python.org> wrote:

One problem I can see with using a fixed offset
tzinfo for localtime is that it might confuse people
when doing date arithmetic.

Yes, this is the issue that I discussed in my first python-ideas/python-dev post. (See msg162631 above.)

>>> d = datetime.localtime() + timedelta(days=7)

While it will give a correct answer as a point in time it
will have the wrong time zone offset if run just before a
daylight saving transition, which could be just as confusing.

I think my latest proposal will fare slightly better in this scenario. I am now propose implementing local timezone info discovery in datetime.astimezone() method. Obtaining local time will now be a little more verbose:

local_time = datetime.now(timezone.utc).astimezone()

but (local_time + timedelta(7)).astimezone() will give you correctly adjusted aware datetime seven days later.

If we also implement astimezone() for naive instances, we can even support naive local time arithmetics: (datetime.now() + timedelta(7)).astimezone() will produce the same time of day as "now" unless it falls into a non-existing hour.

I'm not sure how you'd solve this without e.g. importing
pytz into the standard library though.

Importing pytz will not help. Pytz fudges the issue by extending tzinfo methods to accept isdst flag, but there is no place in datetime to store this flag. Doing time calculations in local time is inherently ambiguous in presence of DST. On the other hand, producing tz aware local time from any unambiguous absolute time specification (UTC, time_t, local time + tz offset, etc.) is a well-defined problem which does not have an adequate solution.

... is a well-defined problem which does not have an adequate solution.

I meant to say "does not have an adequate solution *in the current datetime module*". I think the enhanced datetime.astimezone() method will solve this problem.

Attached patch implements astimezone() default in both Python and C.

New changeset 88a5f2730579 by Alexander Belopolsky in branch 'default':
Issue bpo-9527: datetime.astimezone() method will now supply a class
http://hg.python.org/cpython/rev/88a5f2730579

New changeset 336c53c1f547 by Alexander Belopolsky in branch 'default':
Issue bpo-9527: datetime.astimezone() method will now supply a class
http://hg.python.org/cpython/rev/336c53c1f547

New changeset a7237f157625 by Alexander Belopolsky in branch 'default':
Issue bpo-9527: Fixes for platforms without tm_zone
http://hg.python.org/cpython/rev/a7237f157625

This patch breaks OpenIndiana buildbots. For instance

http://www.python.org/dev/buildbot/all/builders/AMD64%20OpenIndiana%203.x/builds/3810/steps/test/logs/stdio

"""
FAIL: test_astimezone_default_eastern (test.datetimetester.TestDateTimeTZ_Pure)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/support.py", line 1139, in inner
    return func(*args, **kwds)
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/datetimetester.py", line 3286, in test_astimezone_default_eastern
    self.assertEqual(local.strftime("%z %Z"), "+0500 EST")
AssertionError: '-0500 EST' != '+0500 EST'
- -0500 EST
? ^
+ +0500 EST
? ^
"""

Working on this. It turns out tm_gmtoff uses the opposite sign to
that of timezone in time.h. For more confusion, consider this:

$ TZ=EST+5 date +%z
-0500

I am rechecking all UTC offset signs.

On Fri, Jun 22, 2012 at 3:36 PM, Jesús Cea Avión <report@bugs.python.org> wrote:

Jesús Cea Avión <jcea@jcea.es> added the comment:

This patch breaks OpenIndiana buildbots. For instance

http://www.python.org/dev/buildbot/all/builders/AMD64%20OpenIndiana%203.x/builds/3810/steps/test/logs/stdio

"""
FAIL: test_astimezone_default_eastern (test.datetimetester.TestDateTimeTZ_Pure)
----------------------------------------------------------------------

> Traceback (most recent call last):
>  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/support.py", line 1139, in inner
>    return func(*args, **kwds)
>  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/datetimetester.py", line 3286, in test_astimezone_default_eastern
>    self.assertEqual(local.strftime("%z %Z"), "+0500 EST")
> AssertionError: '-0500 EST' != '+0500 EST'
> - -0500 EST
> ? ^
> + +0500 EST
> ? ^
> """
>
>

nosy: +jcea

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue9527\>

New changeset 0f0e3ec22fce by Alexander Belopolsky in branch 'default':
Issue bpo-9527: tm_gmtoff has 'correct' sign.
http://hg.python.org/cpython/rev/0f0e3ec22fce

Navigation Menu

Repository navigation

Add aware local time support to datetime module #53736

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add aware local time support to datetime module #53736

Description

Activity

abalkin commented on Aug 6, 2010

abalkin commented on Aug 6, 2010

abalkin commented on Oct 14, 2010

abalkin commented on Jan 14, 2011

abalkin commented on Jan 14, 2011

rhettinger commented on Jan 30, 2011

abalkin commented on Jan 30, 2011

bitdancer commented on May 5, 2011

ncoghlan commented on Feb 22, 2012

python-dev commented on May 26, 2012

abalkin commented on Jun 11, 2012

abalkin commented on Jun 12, 2012

jamesh commented on Jun 12, 2012

abalkin commented on Jun 19, 2012

abalkin commented on Jun 19, 2012

abalkin commented on Jun 20, 2012

python-dev commented on Jun 22, 2012

python-dev commented on Jun 22, 2012

jcea commented on Jun 22, 2012

abalkin commented on Jun 22, 2012

python-dev commented on Jun 22, 2012

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions