Repository navigation
datetime.strptime(dt.strftime("%c"), "%c")) fails when year is <1000. #124529
Description
Activity
The year for datetime.datetime must be and is allowed to be anything in range
MINYEAR <= year <= MAXYEAR, which is 1 <= year <= 9999. I expect that the format functions should handle any legal date.Reacted by Jan KaliszewskiConsidering these results:
>>> datetime(999, 1, 1).strftime("%c") 'Tue Jan 1 00:00:00 999' >>> datetime.strptime("Tue Jan 1 00:00:00 999", "%c") # as from strftime() above => the error described above [snip] ValueError: time data 'Tue Jan 1 00:00:00 999' does not match format '%c' >>> datetime.strptime("Tue Jan 1 00:00:00 999", "%c") # adding 0 before 999 to have 4-digit width year => success datetime.datetime(999, 1, 1, 0, 0)
...and the following fragment of the docs (https://docs.python.org/3/library/datetime.html#technical-detail):
- The strptime() method can parse years in the full [1, 9999] range, but years < 1000 must be zero-filled to 4-digit width.
...I am not sure if the proviso that years < 1000 must be zero-filled to 4-digit width intentionally covers also this case.One could argue that it does, and there is nothing to fix here.Another person, however, could argue that:
- (1) it does not, as here we deal with a locale-specific way of formatting;
- (2) successful round-trip behavior is an expected property, and dropping it would be surprising.
What do you think?
[EDIT] The quoted note refers to the
%Yformat code, not to the%cone. So I believe that that imaginary Another person would be right. :)PS It seems that for
time.{strftime,strptime}()the behavior is the same (as, apparently,time.strptime()uses the same implementation from_strptime):$ ./python Python 3.14.0a0 (heads/main:a4d1fdfb15, Sep 26 2024, 22:47:21) [GCC 11.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import time >>> t_tuple = time.strptime("Tue Jan 1 00:00:00 0999", '%c') >>> t_tuple time.struct_time(tm_year=999, tm_mon=1, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=1, tm_yday=1, tm_isdst=-1) >>> time.strftime('%c', t_tuple) 'Tue Jan 1 00:00:00 999' >>> time.strptime(_, '%c') Traceback (most recent call last): File "<python-input-4>", line 1, in <module> time.strptime(_, '%c') ~~~~~~~~~~~~~^^^^^^^^^ File "/home/zuo/cpython/Lib/_strptime.py", line 567, in _strptime_time tt = _strptime(data_string, format)[0] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^ File "/home/zuo/cpython/Lib/_strptime.py", line 352, in _strptime raise ValueError("time data %r does not match format %r" % (data_string, format)) ValueError: time data 'Tue Jan 1 00:00:00 999' does not match format '%c'
Hypothesis
It seems that the source of the problem is that (at least typically – for the
C.UTF-8locale and at least some others, e.g.pl_PL.UTF-8; yet, it seems that also for any other locales...):datetime.datetime.strftime()– when%cis used to format a date+time – does not use thedatetime's way of formatting%Y(which would result in a 4-digit year, with leading zeros for year < 1000), but returns a string that contains the year number with minimum count of digits needed to represent that number (i.e., less than 4 for year < 1000).
...whereas...
datetime.datetime.strptime()– when%cis used to parse a date+time – uses, to parse the year fragment, an%Y-specific regex (see the_strptimemodule...) which requires that the year number has exactly 4 digits.
Observation
I checked that:
(1) When formatting that example year 999, the results are:
Function/Method For "%c"For "%Y"time.strftime()"999""999"datetime.datetime.strftime()"999""0999"[sic!]Conclusion:
datetime.datetime.strftime()'s%cformatting behaves liketime.strftime(), therefore it is not based ondatetime.datetime.strftime()'s formatting of%Y.(2) When parsing that example year
999(as well as, e.g.,9) – both as a part of full date (%c) and alone (%Y) – only the 4-digit year format is accepted. Smaller numbers of digits always cause the sameValueErrorfrom_strptime(whose machinery, as noted above, even for%cuses the%Y-specific stuff...).Possible fix
In the
_strptimemodule's machinery (which is used bydatetime.datetime.strptime()andtime.strptime()): decouple the%c's parsing regex from the%Y's one, making the former more liberal (accepting also 1, 2 or 3 digits in the year number).
[The fix implementation would be made in the_strptimemodule,probably somewhere ininLocaleTime.__calc_date_time()/TimeRE.__init__()...TimeRE's__init__()andpattern()](Another theoretically possible variant: just make the
%Y's regex more liberal – however that seems too disruptive...)I'd happy to implement the fix – if you decide that this should be fixed.
No issue on my Macbook laptop
Python 3.14.0a0 (heads/main:162d152146a, Sep 25 2024, 10:45:28) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from datetime import datetime >>> datetime.strptime(datetime(1000, 1, 1).strftime("%c"), "%c") datetime.datetime(1000, 1, 1, 0, 0) >>> datetime.strptime(datetime(999, 1, 1).strftime("%c"), "%c") datetime.datetime(999, 1, 1, 0, 0) >>>Could you please check what string is returned on you system from the following call?
>>> datetime(999, 1, 1).strftime("%c")
Thanx :)
PS My guess is that, for your locale, a
%c-formatted date+time includes a 2-digit year variant (instead of the 4-digit one).@zuo I just tried it just now
Python 3.14.0a0 (heads/main:162d152146a, Sep 25 2024, 10:45:28) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from datetime import datetime >>> datetime(999, 1, 1).strftime("%c") 'Tue Jan 1 00:00:00 0999'Thank you!
Yeah, that leading zero your platform/locale provides makes
strftime's%cformat digestible bystrptimeon your system. Apparently, that's not the case for Linux family [EDIT: or, probably, more generally – forglibc]. :-/Anyway, now it's quite clear for me what the fix should be.
Reacted by MariattaProof of concept:
diff --git a/Lib/_strptime.py b/Lib/_strptime.py index a3f8bb544d..6a2527b75c 100644 --- a/Lib/_strptime.py +++ b/Lib/_strptime.py @@ -213,8 +213,10 @@ def __init__(self, locale_time=None): 'Z'), '%': '%'}) base.__setitem__('W', base.__getitem__('U').replace('U', 'W')) - base.__setitem__('c', self.pattern(self.locale_time.LC_date_time)) - base.__setitem__('x', self.pattern(self.locale_time.LC_date)) + base.__setitem__( + 'c', self.__pattern_with_lax_year(self.locale_time.LC_date_time)) + base.__setitem__( + 'x', self.__pattern_with_lax_year(self.locale_time.LC_date)) base.__setitem__('X', self.pattern(self.locale_time.LC_time)) def __seqToRE(self, to_convert, directive): @@ -236,6 +238,21 @@ def __seqToRE(self, to_convert, directive): regex = '(?P<%s>%s' % (directive, regex) return '%s)' % regex + def __pattern_with_lax_year(self, format): + """Like pattern(), but making %y and %Y accept also fewer digits. + + Necessary to ensure that strptime() is able to parse strftime()'s + output when the %c or %x format code is used -- considering that + for some locales/platforms (e.g., 'C.UTF-8' on Linux), formatting + with either %c or %x may cause year numbers, if a number is small, + to have fewer digits than usual (e.g., '999' instead of `0999', or + '9' instead of '0009' or '09'). + """ + pattern = self.pattern(format) + pattern = pattern.replace(self['y'], r"(?P<y>\d{1,2})") + pattern = pattern.replace(self['Y'], r"(?P<Y>\d{1,4})") + return pattern + def pattern(self, format): """Return regex pattern for the format string.
[EDIT] After applying the above patch, the error does not occur anymore:
>>> import time >>> t_tuple = time.strptime("Tue Jan 1 00:00:00 0999", '%c') >>> t_tuple time.struct_time(tm_year=999, tm_mon=1, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=1, tm_yday=1, tm_isdst=-1) >>> time.strftime('%c', t_tuple) 'Tue Jan 1 00:00:00 999' >>> time.strptime(_, '%c') time.struct_time(tm_year=999, tm_mon=1, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=1, tm_yday=1, tm_isdst=-1) >>> >>> from datetime import datetime >>> datetime(999, 1, 1).strftime('%c') 'Tue Jan 1 00:00:00 999' >>> datetime.strptime(_, '%c') datetime.datetime(999, 1, 1, 0, 0)
@pganssle @terryjreedy @Mariatta @serhiy-storchaka
OK, I'd like to propose the fix, implemented in the linked PR #124778 – considering that:
- The status quo – the error-causing discrepancy between
strftimeandstrptime(in respect to%cand%x), concerning yearnumbersnumber ranges explicitly documented as valid (i.e., within the range 1..9999) – is suboptimal and it seems that, as @terryjreedystatedseems to suggest above, it should be considered a bug. The fact that the presence of the problematic behavior is platform-dependent makes the state of affairs even less consistent. - Altering
strftimeto produce only zero-padded year representations (when it comes to the%c/%xformat codes) does not seem like a good idea to me, because:- the platform-dependent behavior of
strftimeis a well-established (and documented) fact of live (contract?); it is not clear, whether (and for whom, considering the wide range of Python uses/applications/platforms...) it would be acceptable or unacceptable to deviate from it – at least when it comes to these particular, locale-dependent, format codes%cand%x[EDIT: I admit that the changes made in gh-120713 and gh-122272 weaken this argument to some extent; but, on the other hand, the composite nature and the unpredictability caused by being locale-dependent makes these two format codes –%cand%x– somewhat different beasts (in this context) than%Y,%G,%C,%F; the latter (%F) is also composite, indeed, but still locale-indepentent and easily explainable in terms of other format codes] - the implementation would be non-trivial (re-parse and alter the output obtained from the underlying platform routines? would it be worth the complexity? what about the performance penalty?).
- the platform-dependent behavior of
- Altering
strptime, in the spirit of the robustness principle, to accept also non-zero-padded year representations – only regarding%cand%x(i.e., not touching%Y/%y/etc.) – seems to be the right solution, considering that:- generally, the platform-independent (Python-devs-only-steered) behavior of
strptimeis a well-established (and documented) fact, so this is the natural place to adjust the stuff; - the compatibility impact seems negligible; [EDIT: indeed, starting to accept less typical and previously unacceptable inputs does pose some risk of parsing garbage as valid data, but – given the composite nature of
%cand%x– isn't that risk minimized enough?] - the implementation is very simple (< 10 SLOC, excluding tests/docstrings/comments – see the PR).
- generally, the platform-independent (Python-devs-only-steered) behavior of
- The status quo – the error-causing discrepancy between
strptime()should be able to parse strings created by other programs, not only Python, so this is a bug, and the only solution is to alterstrptime().Reacted by Jan KaliszewskiI do not think that #124778 is a right solution. We should fix %Y, %G and maybe %y. This will automatically fix %c and %x. And I consider this a bugfix which should be backported.
I do not think that #124778 is a right solution. We should fix %Y, %G and maybe %y. This will automatically fix %c and %x. And I consider this a bugfix which should be backported.
But wouldn't making
%Y/%G/%yaccept even a 1-digit year be too aggressive change? I have no hard data, but can imagine quite easily, that a non-negligible number of users may rely, e.g., on%Yto distinguish valid dates (e.g.,2024/09/24) from garbage or highly ambiguous inputs (e.g.24/09/24or1/10/1).My proposal in #124778 is similar, just much more conservative (as
%c/%xare always supposed to be used in the realm of a particular locale; that, I believe – together with the fact that internally they are always composed of several fields – makes the change much safer), and limited just to those cases in which thestrftime/strptimeparity cannot [or at least not easily] be gained by fixing thestrftime's "side of the equation" (which has already been done for%Y/%Gin gh-120713).Let me also emphasize that
strptime's%Yand%yare explicitly documented as requiring the leading zeros.(And there are no such statements in the docs when it comes to
%cand%x; only that we deal with a locale’s appropriate representation.)For the record, I'd be happy to continue working on this. :)
The approach I proposed (and drafted in the linked PR gh-124778) is conservative and focuses just on the problem originally described by @pganssle. I believe my arguments are worth considering, so I hope for the continuation of the discussion and, ultimately, reaching some conclusion (either way). Of course, I understand and respect that it must wait until a core developer has the time and is willing to take this on.
PS The issue can still be reproduced on the current
main.
Bug report
Bug description:
Discovered this when adding some hypothesis tests for
strptime/strftime. I doubt this is a real problem anyone is going to have in the real world, but maybe.I do not know if this is locale-specific or OS specific.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs
_strptimeto make%c/%xaccept a year with fewer digits #124778