Repository navigation
os.stat(): add new fields to get timestamps as Decimal objects with nanosecond resolution #55666
Description
| BPO | 11457 |
|---|---|
| Nosy | @loewis, @rhettinger, @jcea, @mdickinson, @abalkin, @gustaebel, @vstinner, @larryhastings, @bitdancer, @skrah |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/rhettinger'
closed_at = <Date 2012-02-26.22:51:27.870>
created_at = <Date 2011-03-10.00:05:27.410>
labels = ['type-feature', 'library']
title = 'os.stat(): add new fields to get timestamps as Decimal objects with nanosecond resolution'
updated_at = <Date 2012-02-26.22:51:27.869>
user = 'https://bugs.python.org/khenriksson'bugs.python.org fields:
activity = <Date 2012-02-26.22:51:27.869>
actor = 'larry'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2012-02-26.22:51:27.870>
closer = 'larry'
components = ['Library (Lib)']
creation = <Date 2011-03-10.00:05:27.410>
creator = 'khenriksson'
dependencies = ['11941']
files = ['23246', '24309', '24321']
hgrepos = []
issue_num = 11457
keywords = ['patch']
message_count = 65.0
messages = ['130478', '130479', '130596', '134642', '137558', '137578', '137580', '137593', '137599', '137600', '137606', '137608', '137877', '137888', '138978', '138979', '138980', '138984', '138987', '139169', '139321', '143573', '143644', '143738', '143739', '143801', '143802', '143803', '143805', '143807', '143811', '143812', '143819', '143820', '143837', '143866', '143867', '143868', '143873', '143881', '143885', '143898', '144543', '144607', '145256', '145262', '145288', '151872', '151873', '151912', '151943', '151987', '151992', '152003', '152004', '152305', '152306', '152314', '152317', '152320', '152322', '152323', '152350', '152355', '154405']
nosy_count = 16.0
nosy_names = ['loewis', 'rhettinger', 'jcea', 'mark.dickinson', 'belopolsky', 'lars.gustaebel', 'vstinner', 'larry', 'nadeem.vawda', 'Arfrever', 'r.david.murray', 'skrah', 'Alexander.Belopolsky', 'rosslagerwall', 'khenriksson', 'ericography']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue11457'
versions = ['Python 3.3']
Activity
khenriksson commentedon Mar 10, 2011
The most recent (issue 7) release of the POSIX standard mandates support for nanosecond precision in certain system calls. For example, the stat structure include a timespec struct for each of mtime, atime, and ctime that provides such nanosecond precision.[1] There is also an futimens call that allows setting the time accurate to the nanosecond.[2] Support for such precision is available at the least on 2.6 Linux kernels.
Currently, the Python float type is used everywhere to express times in a single value (such as the result from os.stat). However, since this is implemented in CPython using type double (and possibly similarly elsewhere) it is impossible to obtain sufficient precision using a float.
Therefore, it would be useful to expose the number of seconds and nanoseconds separately
to allow full precision. Perhaps adding these values as additional members to the return value from os.stat would be most useful, something like .st_atimensec that Linux sometimes uses, or else just follow the POSIX standard to include a sub-struct.
This is important for example with the tarfile module with the pax tar format. The POSIX tar standard[3] mandates storing the mtime in the extended header (if it is not an integer) with as much precision as is available in the underlying file system, and likewise to restore this time properly upon extraction. Currently this is not possible.
khenriksson commentedon Mar 10, 2011
Also, a new function similar to os.utime would be needed as well, perhaps something named like os.utimens. This would be needed to allow setting times with nanosecond precision.
abalkin commentedon Mar 11, 2011
See also bpo-10812 which implements os.futimens().
rosslagerwall commentedon Apr 28, 2011
Closed bpo-11941 as a duplicate of this.
bitdancer commentedon Jun 3, 2011
The mailbox module would benefit from having this precision available.
loewis commentedon Jun 3, 2011
I suggest that rather than using composite time stamps, decimal.Decimal is used to represent high-precision time in Python.
On input to os.utime, the function could just polymorphically accept Decimal, and try its best.
I see three approaches that preserve compatibility for stat (plus the incompatible one of just changing the field types of struct stat):
This would be appropriate if the ultimate goal is to eventually change
the fields in an incompatible way in Python 4.
Arfrever commentedon Jun 3, 2011
os.utimensat() and os.futimens() already exist since Python 3.3 and require 2-tuples (or None) as second and third argument.
(utime() is deprecated since POSIX 2008: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/utime.h.html)
(Changes specific to os.stat() are discussed in issue bpo-11941.)
abalkin commentedon Jun 3, 2011
On Fri, Jun 3, 2011 at 3:57 PM, Martin v. Löwis <report@bugs.python.org> wrote:
..
I support this idea in theory, but as long as decimal is implemented
in Python, os module should probably expose a low level (tuple-based?)
interface and a higher level module would provide Decimal-based
high-precision time.
BTW, what is the status of cdecimal?
loewis commentedon Jun 3, 2011
Am 03.06.2011 22:11, schrieb Arfrever Frehtes Taifersar Arahesis:
"Already since 3.3" means "they don't exist yet". I.e. it isn't too late
to change them.
This is a case where I think Python shouldn't follow POSIX deprecation.
In C, you need to change the function name to change the parameter
types; not so in Python.
loewis commentedon Jun 3, 2011
Can you explain why you think so? I fail to see the connection.
abalkin commentedon Jun 3, 2011
On Fri, Jun 3, 2011 at 6:13 PM, Martin v. Löwis <report@bugs.python.org> wrote:
..
One reason is the desire to avoid loading Python module from a
C-module. I understand that this ship has already left the port with
larger and larger portions of stdlib being implemented in Python, but
doing that in a basic module such as os (or rather posix) is likely to
cause more problems than what we have in other similar situation. For
example, strptime is implemented in a Python module loaded by time and
datetime implemented in C. This works, but at a cost of extreme
trickery in the test suit and similar problems encountered by
sophisticated applications. As far as I remember, some multi-threding
issues have never been resolved.
loewis commentedon Jun 3, 2011
This desire is indeed no guidance for Python development; the opposite
is the case. The only exception may be bootstrapping issues, which I
claim are irrelevant in this case.
abalkin commentedon Jun 7, 2011
On Fri, Jun 3, 2011 at 6:52 PM, Martin v. Löwis <report@bugs.python.org> wrote:
..
Can you elaborate on this? I did notice the current trend of mixing
software layers and welcoming circular dependencies in Python stdlib,
but I am not sure this is a good thing. In the good old times imports
inside functions where frowned upon. (And for many good reasons.)
Imports from inside C functions seem to be even worse. Tricks like
this greatly reduce understandability of the code. The import
statements at the top of the module tell a great deal about what the
module can and cannot do. When modules can be imported at will as a
side-effect of innocuous looking functions (time.strptime is my
personal pet peeve), analysis of the programs becomes much more
difficult.
It is hard to tell without attempting an implementation, but my
intuition is exactly the opposite. I believe parts of the import
mechanism have been implemented in Python and it seems to me that
os.stat() may need to be available before decimal can be imported.
41 remaining items
rhettinger commentedon Jan 26, 2012
Have you researched how other languages plan to expose sub-millisecond times? The isn't an API that will get points for originality. Also, it needs to be an API that is time efficient (many scripts use os.stat() frequently to scan files for changes and that check needs to be fast).
Please do not "just check it in".
vstinner commentedon Jan 26, 2012
Using decimal timestamps should be an option, float timestamps must
remain the default.
larryhastings commentedon Jan 26, 2012
Victor: I *think* Raymond's comments were directed at my patch, not yours.
vstinner commentedon Jan 30, 2012
I don't like the idea of adding new fields to os.stat() *by default* because it may break backward compatibility. And if the new fields are decimal.Decimal objects, the module has to be imported and it means that any call to os.stat() would be slower just to provide timestamps with a finer resolution: this is not acceptable if you just want to check if a file exists.
In the issue bpo-13882, I propose to add a format argument to functions getting time (time.time(), time.clock(), etc.). My patch doesn't change the default type, but add a "decimal" format to get time as a decimal.Decimal object.
The option (format) value is a string to be able to add other formats without having to change the API later:
For os.stat(), the optional argument can be called "timestamp".
So if you want timestamps in the best available resolution, use timestamp="decimal". If you prefer the datetime API, use timestamp="datetime". If you don't care of timestamps, just call os.stat() without setting the timestamp option ;-)
We might add a registry to add user-defined types, but the "tuple" format should be enough. (I don't know if we need to expose the low level "tuple" format.)
Arfrever commentedon Jan 30, 2012
I think that one of available types of time values returned by os.stat() should allow to directly pass these values to os.futimens() and os.utimensat(), which expect (time_sec, time_nsec) tuples.
vstinner commentedon Jan 30, 2012
If we choose to give the possibility to get decimal.Decimal objects,
we should also patch some functions to support Decimal objects, like
datetime.datetime.fromtimestamp() (today, it "just works", because a
Decimal object can be converted to float). We may accept Decimal in
os.futimens() and os.utimensat(), but I am not sure for this
particular case.
To come back to my format solution, we can also support classical C
structures to interact with C functions (in Python or more directly
using ctypes):
Or we may introduce conversion functions from other types like float,
Decimal or another type.
vstinner commentedon Jan 30, 2012
Oh, I realized that these two functions were added to Python 3.3, so
it is not too late to change their API. I would prefer to limit the
number of timestamp formats: Python 3.2 has float and datetime, I (and
Martin) propose to add Decimal to Python 3.3 (to get nanosecond
resolution). (sec, nsec) is a new format, except if Python 3.2 has
already functions expecting such tuple?
I know that the underlying C function expects a timespec structure,
but Python can try to use a higher level API, isn't it?
Decimal is more practical than a tuple because you can just write :
t2-t1 to compute a time delta. Decimal has other advantages (read the
issue for the full list ;-)).
Arfrever commentedon Jan 30, 2012
(secs, nsecs) tuples are more practical in performance-critical applications (e.g. synchronization of timestamps between 2 trees with large number of files).
vstinner commentedon Jan 30, 2012
This is also why I propose an argument to choose the format: everyone
has a different use case and use cases are incompatible. Tuples are
quick to create but has not a practical API, datetime has a nice API
but also issues listed before by Martin (and don't support nanosecond
resolution currently), Decimal has a nice API but is expensive to
create, etc.
bitdancer commentedon Jan 30, 2012
There is also the fact that we have traditionally exposed thin wrappers around posix functions (and then were practical provided Windows emulations). We aren't 100% consistent about this, but we are pretty consistent about it.
loewis commentedon Jan 30, 2012
I agree entirely.
vstinner commentedon Jan 30, 2012
I attached a more complete patch to the issue bpo-13882: it adds an optional timestamp format to os.stat(), os.lstat(), os.fstat(), os.fstatat().
Examples:
larryhastings commentedon Feb 26, 2012
Given Guido's rejection of PEP-410, this won't happen, so I'm closing this bug. Our BFDL has specifically rejected any of the complicated representations; he ruled that all we need are new _ns fields representing the time in nanoseconds, and to accept a "ns=" argument for os.utime and its ilk. Please see bug bpo-14127 for discussion of that change.