Repository navigation
imaplib: Internaldate2tuple produces wrong result if date is near a DST change #55150
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/abalkin'
closed_at = <Date 2014-06-29.20:25:14.394>
created_at = <Date 2011-01-18.23:41:44.481>
labels = ['type-bug', 'library']
title = 'imaplib: Internaldate2tuple produces wrong result if date is near a DST change'
updated_at = <Date 2014-06-29.20:25:14.392>
user = 'https://bugs.python.org/lavajoe'bugs.python.org fields:
activity = <Date 2014-06-29.20:25:14.392>
actor = 'r.david.murray'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2014-06-29.20:25:14.394>
closer = 'r.david.murray'
components = ['Library (Lib)']
creation = <Date 2011-01-18.23:41:44.481>
creator = 'lavajoe'
dependencies = []
files = ['20446', '20447', '20615', '25285', '25297', '26100']
hgrepos = []
issue_num = 10941
keywords = ['patch']
message_count = 27.0
messages = ['126503', '126573', '127490', '127502', '127507', '127508', '158645', '158646', '158661', '158662', '158663', '158664', '158665', '158733', '158798', '158901', '159105', '159112', '159144', '159146', '159148', '159151', '159649', '163505', '163510', '221896', '221897']
nosy_count = 5.0
nosy_names = ['georg.brandl', 'belopolsky', 'r.david.murray', 'lavajoe', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue10941'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4']
Activity
lavajoe commentedon Jan 18, 2011
DST is not handled correctly. Specifically, when the input date/time, ignoring the TZ offset (and treated as if it is local time) is on the other side of the DST changeover from the actual local time represented, the result will be off by one hour. This can happen, e.g., when the input date/time is actually UTC (offset +0000).
This is because the check for DST is done after converting the time into a local time tuple, thereby treating as real local time. This can be corrected by keeping the time in real UTC (by using calendar.timegm() instead of checking the DST flag) until the final conversion to local time.
Here is an example: Run the following two dates, that represent exactly the same time, through Internaldate2tuple:
'25 (INTERNALDATE "01-Apr-2000 19:02:23 -0700")'
'101 (INTERNALDATE "02-Apr-2000 02:02:23 +0000")'
Note that a variant of this issue (but identifying it as a different problem) was addressed in a similar way in an old post I found by Colin Brown in 2004 (http://www.velocityreviews.com/forums/t336162-imaplib-function-bug.html).
Patch also adds unit test to check the above example dates. Python 3 version of patch assumes patch from bpo-10939 has been applied.
Patch also corrects code python doc that currently states time is UT, not local (see bpo-10934).
abalkin commentedon Jan 19, 2011
I agree that this is a bug and the solution is right, I am not sure this should be applied during RC period. The bug has been present and known for a long time, so affected parties probably have a work-around in place already.
lavajoe commentedon Jan 29, 2011
Here is a new patch that adds a test to the tests committed for bpo-10939.
This new test case is needed to trigger the DST issue. This test is not timezone-dependent (in the sense that one does not have to have a specific timezone set for it to work), but it does depend on the DST change happening at 02:00 on the first Sunday in April, 2000. This is the case for most of the United States, but to make it work everywhere, I have added code that sets the TZ to one that works, and then reverts the temporary TZ change after the test.
The bug is triggered when the time described in the internal date string, ignoring the time offset, is between 02:00 and 03:00. This is because the raw time in the string is interpreted as local time, and local times in this range are basically invalid, since time advances to 03:00 when 02:00 is reached because of the DST change. This causes the final result to be off by one hour.
[This patch is only for Python 3]
abalkin commentedon Jan 30, 2011
On Sat, Jan 29, 2011 at 4:45 PM, Joe Peterson <report@bugs.python.org> wrote:
..
Your test will not restore TZ if it fails. This is not nice. To fix
that, code restoring the TZ should go into a finally clause. Even
better, set TZ in a setUp() method and restore it in a tearDown()
method of the test case class. Ultimately, if you want to be fancy,
take a look at the @run_with_locale decorator defined in
Lib/test/support.py. We may have more than one use for @run_with_tz.
lavajoe commentedon Jan 30, 2011
I like the idea of adding the decorator. New patch added.
lavajoe commentedon Jan 30, 2011
[just carrying over some info from bpo-10939 that is related to this issue]
Here is another manifestation of this issue, related to the local time assumption, but not to DST, per se:
Here is the definition for Europe/London in the unix tz data:
# Zone NAME GMTOFF RULES FORMAT [UNTIL]
Zone Europe/London -0:01:15 - LMT 1847 Dec 1 0:00s
0:00 GB-Eire %s 1968 Oct 27
1:00 - BST 1971 Oct 31 2:00u
0:00 GB-Eire %s 1996
0:00 EU GMT/BST
So London's local time was always 1 hour ahead of UTC (BST time) from 1968 Oct 27 until 1971 Oct 31 2:00u.
Because of this, Internaldate2tuple() returns the wrong local time (00:00 rather than [the correct] 01:00) in its tuple when input is "01-Jan-1970 00:00:00 +0000" (the epoch).
lavajoe commentedon Apr 18, 2012
It's been over a year since any activity on this bug, and it is still in the "patch review" stage. Any news? Anything else I can provide to help to get this moved to the next stage? Thanks!
bitdancer commentedon Apr 18, 2012
Someone needs to find time to review it.
You could try recruiting reviewers on python-list. Anyone can do a review. Obviously the more knowledge they have in this area the better, but any good review is likely to move the issue along.
lavajoe commentedon Apr 18, 2012
Ah. I figured that one of the Python devs would be who would review it. Is this the normal path for bugs? Not to question the process, but I find it surprising, since I would think that it would cause a lot of bugs to stall out. Also, I had no idea it was the submitter's responsibility to recruit reviewers. And I am not quite sure how to go about this. I do care about this bug (as it's pretty serious that it returns wrong results), so I am willing to do it. Any wiki page or anything that describes the recruiting process or has more info?
AlexanderBelopolsky commentedon Apr 18, 2012
I'll review your patch.
On Apr 18, 2012, at 4:53 PM, Joe Peterson <report@bugs.python.org> wrote:
bitdancer commentedon Apr 18, 2012
I see Alexander is going to take care of this. But to clarify what I suggested for your information:
In an ideal world it would be a committer doing the patch review, followed by a checkin. But in the real world there aren't enough of us with enough time to get to all the bugs with patches. You asked how to move it along and I suggested one way: get someone else to do a review. I wouldn't say that the submitter recruiting a reviewer was a "normal" process, but it is a way to get bugs unstuck. And we get reviews from non-committers frequently (it is a step along the path to becoming a committer...quality reviews are as important as quality patches).
I don't think there's anything in the devguide about this particular nuance.
lavajoe commentedon Apr 18, 2012
Thanks!! :)
lavajoe commentedon Apr 18, 2012
David, I understand - thanks for the details. Hopefully I can return the favor and review one in the future.
2 remaining items
lavajoe commentedon Apr 21, 2012
I have now included a patch for 2.7. Here are the two latest patches:
Python 2: issue10941_python2.diff
Python 3: issue10941_python3.diff
abalkin commentedon Apr 24, 2012
I still get conflicts:
Are you sure your patch is against the head of the hg repository?
lavajoe commentedon Apr 24, 2012
The one you tried is the old patch. Here are the two new patches. Use these:
Python 2: issue10941_python2.diff
Python 3: issue10941_python3.diff
abalkin commentedon Apr 24, 2012
My bad. For some reason I assumed that the latest patch would be at the top of the files list.
David, the patch is good. Should I go ahead and commit?
lavajoe commentedon Apr 24, 2012
Great to hear, Alexander. Thanks for reviewing! Is it also possible to get the pyhton2.7 version one in?
abalkin commentedon Apr 24, 2012
On Tue, Apr 24, 2012 at 10:42 AM, Joe Peterson <report@bugs.python.org> wrote:
..
I don't see any reason not to. This is a bug fix and should go into a
maintenance release. I will wait to hear from David, who is the
maintainer of this module, though.
bitdancer commentedon Apr 24, 2012
If you are satisfied with the time logic then yes, please apply it. I suspect that the number of people using the code and not aware of the problem (or not caring enough to do anything about it) exceeds the number aware of it who have coded a workaround, so I think putting it in 2.7 (and 3.2) is better than not doing so.
python-dev commentedon Apr 29, 2012
New changeset 90b9781ccb5f by Alexander Belopolsky in branch '3.2':
Issue bpo-10941: Fix imaplib.Internaldate2tuple to produce correct result near
http://hg.python.org/cpython/rev/90b9781ccb5f
New changeset 6e029b6c142a by Alexander Belopolsky in branch 'default':
Issue bpo-10941: Fix imaplib.Internaldate2tuple to produce correct result near
http://hg.python.org/cpython/rev/6e029b6c142a
abalkin commentedon Jun 22, 2012
I updated the fix to take advantage of resolved bpo-9527. I also noticed and fixed another bug: Internaldate2tuple was using locale-dependent %b directive for strftime.
abalkin commentedon Jun 23, 2012
The latest patch belongs to bpo-11024.
abalkin commentedon Jun 29, 2014
David,
Is there anything left to do here that is not covered by bpo-11024?
bitdancer commentedon Jun 29, 2014
Not that I can see.