Repository navigation
Improve pickle format for timezone aware datetime instances #53297
Description
| BPO | 9051 |
|---|---|
| Nosy | @tim-one, @freddrake, @mdickinson, @abalkin, @pitrou, @vstinner, @avassalotti, @ned-deily, @serhiy-storchaka |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/abalkin'
closed_at = <Date 2017-03-06.17:30:07.984>
created_at = <Date 2010-06-21.20:15:07.758>
labels = ['type-feature']
title = 'Improve pickle format for timezone aware datetime instances'
updated_at = <Date 2017-03-06.17:30:07.973>
user = 'https://github.com/abalkin'bugs.python.org fields:
activity = <Date 2017-03-06.17:30:07.973>
actor = 'belopolsky'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2017-03-06.17:30:07.984>
closer = 'belopolsky'
components = []
creation = <Date 2010-06-21.20:15:07.758>
creator = 'belopolsky'
dependencies = ['9183']
files = ['17739', '35804', '40852']
hgrepos = []
issue_num = 9051
keywords = ['patch']
message_count = 19.0
messages = ['108313', '108408', '108492', '108608', '108611', '108619', '221920', '221933', '221934', '221935', '221936', '251342', '251668', '251677', '251812', '251814', '253402', '254721', '254724']
nosy_count = 10.0
nosy_names = ['tim.peters', 'fdrake', 'mark.dickinson', 'belopolsky', 'pitrou', 'vstinner', 'alexandre.vassalotti', 'ned.deily', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue9051'
versions = ['Python 3.6']
Activity
abalkin commentedon Jun 21, 2010
>>> s = pickle.dumps(timezone.utc) >>> pickle.loads(s) Traceback (most recent call last): .. TypeError: ("Required argument 'offset' (pos 1) not found", <class 'datetime.timezone'>, ())abalkin commentedon Jun 22, 2010
Python version is fixed in sandbox and committed in r82154.
abalkin commentedon Jun 23, 2010
Committed in r82184. Leaving the issue open pending a more thorough review of pickling in datetime module.
abalkin commentedon Jun 25, 2010
The datetime module provides compact pickled representation for date, datetime, time and timedelta instances:
type: size
date: 34
datetime: 44
time: 36
timedelta: 37
On the other hand, current pickle size for timezone is 64 and the size of an aware datetime instance is 105 bytes.
Since stability is important for pickle format, the best format should be developed before the first release to include timezone class.
[-]Cannot pickle timezone instances[/-][+]Improve pickle format for aware datetime instances[/+]freddrake commentedon Jun 25, 2010
As part of this, we should ensure references to common timezones, like
UTC, only create references to a single instance rather than filling
memory with multiple instances.
One consequence of this is that shared instances should probably be immutable.
abalkin commentedon Jun 25, 2010
I am attaching a python prototype implementing interned UTC instance pickling. The patch is against sandbox revision r82218 of datetime.py.
Note that the pickling protocol requires that an instance or factory function is defined at the module level.
The pickle size saving is substantial:
but there is still room for improvement:
>>> len(dumps(datetime.now())) 44I do feel, however, that further improvements will see diminishing returns.
pitrou commentedon Jun 29, 2014
Your latest patch doesn't have a review link. Would you like to regenerate it against the latest default?
abalkin commentedon Jun 30, 2014
I updated the patch. Please note that it only includes python code, so you need to suppress _datetime acceleration to test:
abalkin commentedon Jun 30, 2014
Hmm. Still no "review" link. Is there something special that I need to do to get it? Use different name, maybe?
ned-deily commentedon Jun 30, 2014
FWIW, I see review links for both of your files: all the way over to the right in the Files section.
abalkin commentedon Jun 30, 2014
Now I do too. I guess they just take time to appear. Note that I unlinked the file that Antoine complained about.
[-]Improve pickle format for aware datetime instances[/-][+]Improve pickle format for timezone aware datetime instances[/+]serhiy-storchaka commentedon Sep 22, 2015
For stability it is better to use public name 'timezone.utc' instead of '_utc'. Dotted names now are supported with all protocols.
Don't worry about this. When you pickle a number of datetime objects with the same timezone, the timezone is pickled only once and for all other datetime objects only a reference is used.
abalkin commentedon Sep 26, 2015
Can you elaborate on the "stability" consideration?
I would like to revisit this issue since we will have some changes made to datetime pickles in the context of PEP-495.
serhiy-storchaka commentedon Sep 27, 2015
For now "_utc" is not a part of API. When use it in pickling, you implicitly make it a part of API, and should support it forever (and force other Python implementations to support it). As far as it looks as implementation detail, there is a risk that the "_utc" variable can be renamed and this will break pickle compatibility.
abalkin commentedon Sep 29, 2015
I think we will still need to maintain _utc global indefinitely to keep the old pickles readable.
On the other hand, it looks like support for qualnames is only available with pickle protocol 4, so I don't see any downside from from storing the full qualname other than an 8-byte increase in the size of the pickle.
I guess my position on this will be +0: I'll accept a patch if someone will submit it, but I am unlikely to do it myself.
serhiy-storchaka commentedon Sep 29, 2015
We have no need to maintain _utc global indefinitely if don't add it.
No, support for qualnames now is available with all pickle protocols.
serhiy-storchaka commentedon Oct 24, 2015
Following patch adds tests for timezone pickling and copying. Note, that timezone.utc identity is preserved.
python-dev commentedon Nov 16, 2015
New changeset e48da8f01316 by Serhiy Storchaka in branch '3.4':
Issue bpo-9051: Added tests for pickling and copying the timezone objects.
https://hg.python.org/cpython/rev/e48da8f01316
New changeset 634905e9628d by Serhiy Storchaka in branch '3.5':
Issue bpo-9051: Added tests for pickling and copying the timezone objects.
https://hg.python.org/cpython/rev/634905e9628d
New changeset 9c31544d6a66 by Serhiy Storchaka in branch 'default':
Issue bpo-9051: Added tests for pickling and copying the timezone objects.
https://hg.python.org/cpython/rev/9c31544d6a66
serhiy-storchaka commentedon Nov 16, 2015
In best case the UTC timezone costs only 2 bytes per datetime instance (BINGET + 1-byte index). In worst case it can cost 5 bytes (LONG_BINGET + 4-bytes index). Saving few bytes for single timezone instance has significant effect only for small pickled data. But in this case it is more advantageous to save a datetime just as a timestamp. I suggest to close this issue.