Repository navigation
[Windows] time: crash on formatting time with de_DE locale #80973
Description
| BPO | 36792 |
|---|---|
| Nosy | @pfmoore, @vstinner, @tjguk, @jkloth, @zware, @eryksun, @zooba, @Zheaoli |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-03-27.05:48:25.978>
created_at = <Date 2019-05-04.12:33:18.656>
labels = ['interpreter-core', '3.7', 'OS-windows']
title = '[Windows] time: crash on formatting time with de_DE locale'
updated_at = <Date 2021-03-27.05:48:25.977>
user = 'https://bugs.python.org/CharlieClark'bugs.python.org fields:
activity = <Date 2021-03-27.05:48:25.977>
actor = 'eryksun'
assignee = 'none'
closed = True
closed_date = <Date 2021-03-27.05:48:25.978>
closer = 'eryksun'
components = ['Interpreter Core', 'Windows']
creation = <Date 2019-05-04.12:33:18.656>
creator = 'CharlieClark'
dependencies = []
files = ['48303', '48306', '48307', '48310', '48324']
hgrepos = []
issue_num = 36792
keywords = []
message_count = 40.0
messages = ['341380', '341436', '341437', '341438', '341442', '341444', '341456', '341457', '341462', '341463', '341470', '341473', '341503', '341526', '341529', '341551', '341552', '341561', '341562', '341636', '341669', '341698', '341754', '341755', '341760', '341762', '341773', '341779', '341781', '341784', '341790', '342045', '342232', '342923', '342924', '342966', '361471', '361473', '361871', '389590']
nosy_count = 12.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'jkloth', 'jeremy.kloth', 'zach.ware', 'eryksun', 'steve.dower', 'Manjusaka', 'CharlieClark', 'Dominik Geldmacher', 'Carsten Fuchs']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36792'
versions = ['Python 3.7']
Activity
CharlieClark commentedon May 4, 2019
Based on a bug report (https://bitbucket.org/openpyxl/openpyxl/issues/1266/locale) from a user of the openpyxl library I've identified a bug in the zipfile module that causes the Python process to crash on Windows. Currently tested with Python 3.7.3 (32-bit on Windows 10).
Sample code
faulthandler fingers line 1757 as the culprit but running this line locally does not cause the crash. The issue seems to be limited to Windows.
Zheaoli commentedon May 5, 2019
I can't reproduce this error on my system by using the same code.
Could you share the system info with us?
Zheaoli commentedon May 5, 2019
Or would you share the Exception with us ?
I guess it's caused by system setting
CharlieClark commentedon May 5, 2019
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] on win32
Zheaoli commentedon May 5, 2019
Same environment.
But I still can not reproduce this exception. I guess maybe it's about the local time or timezone problem. I will find a way to figure it out
CharlieClark commentedon May 5, 2019
That's what we thought when we looked at it, but as I said, I couldn't reproduce it with just the
timecall or theZInfoinstantiation, so something odd is happening. I do have a German version of Windows as I suspect the original reporter does. You'd think that sort of thing wouldn't matter…DominikGeldmacher commentedon May 5, 2019
I can reproduce it on Python 3.7.3 german Windows10 enterprise
Windows fatal exception: code 0xc0000374
Current thread 0x00003bc8 (most recent call first):
File "C:\Python37\lib\zipfile.py", line 1757 in writestr
Zheaoli commentedon May 5, 2019
Hi Dominik Geldmacher
May I get your system version? 1809 or 1903?
I guess maybe it's microsoft error
DominikGeldmacher commentedon May 5, 2019
Enterprise 1809 and Professional 1803 (german localized) both reproduce the issue
Zheaoli commentedon May 5, 2019
copy that
I will reset my locale setting to figure it out
CharlieClark commentedon May 5, 2019
winver tells me I have 1809. I'm only using Windows in a VM so I'm not that familiar with its innards.
Also get the error with WinPython 3.6:
Windows fatal exception: code 0xc0000374
Current thread 0x000010c0 (most recent call first):
File "C:\Users\charlieclark\WPy64-3680\python-3.6.8.amd64\lib\zipfile.py", line 1658 in writestr
File "<stdin>", line 1 in <module>
jkloth commentedon May 5, 2019
Related to issue bpo-36319
16 remaining items
CharlieClark commentedon May 7, 2019
And this is the result.
old locale: C
count: 28
value: Mitteleuropäische Sommerzeit
new locale: de_DE
count: -1
value:
Windows fatal exception: code 0xc0000374
Looks like
is the culprit.
jkloth commentedon May 7, 2019
Final test, this time, no Python what so ever. I've added a zip containing a simple C program (source and .exe) that performs the same test.
The output should be similar to:
--------
The current locale is now: C
The time zone is: 'Mountain Daylight Time' (22 characters)
--------
The updated locale is now: de_DE
The time zone is: 'Mountain Daylight Time' (22 characters)
--------
If this crashes, the problem is then within the CRT itself :(
CharlieClark commentedon May 7, 2019
This is the result
\bpo-36792>test.exe
--------
The current locale is now: C
The time zone is: 'Mitteleuropõische Sommerzeit' (28 characters)
--------
The updated locale is now: de_DE
The time zone is: '' (-1 characters)
--------
NB something is wrong with that string. Should be Mitteleuropäsiche Sommerzeit but that could just be an encoding thing.
jkloth commentedon May 7, 2019
Thanks again! I will have some more tests for you to try tomorrow as I am out of time for today.
I'm currently of the belief that there is something Python is going to have to do to work around an issue within the CRT, but more testing will prove that theory.
jkloth commentedon May 10, 2019
I've added another test executable (bpo-36792-2.zip) which should bring some insight into where things are going wrong. Please run and post the results.
jkloth commentedon May 12, 2019
I have managed to setup a VM that can reproduce the error. Unfortunately, the error (heap corruption) is coming from within the UCRT. Attempting to work around that, I came across another error in the UCRT.
Due to these errors in all available UCRT versions, the only available solution is to re-implement %Z handling in Python's implementation of strftime().
@steve.dower: If you could forward the following information to the interested parties for faster response, it would be great.
Everyone else, what follows contains UCRT internal details which is of no interest to Python.
In wcsftime() [ucrt\time\wcsftime.cpp, line 1018], the call to _mbstowcs_s_l() doesn't handle the error case of EILSEQ. This error happens when the tzname (originally set by __tzset() from the Unicode name in the registry encoded to the ACP) contains an undecodable character. In the 'de_DE' case, \u00E4. _mbstowcs_s_l returns without changing
wnum(which is 0). The 'Z' case then blindly subtracts 1 (wnum=-1) and advances the string pointer, now pointing before the start when the format is just L"%Z" (like Python has). Now when wcsftime() is returning it stores a '\0' at that current position thus causing a heap corruption.Ok, no deal breaker, we can just update the _tzname array when we call setlocale(). Alas, another error :( If _tzset() is called a second time, it fails when encoding the Unicode name due to an uninitialized variable. In tzset_from_system_nolock() [ucrt\time\tzset.cpp, line 158], the function __acrt_WideCharToMultiByte() [appcrt\locale\widechartomultibyte.cpp] does not set the
lpUsedDefaultCharflag when FALSE. This is only an issue, it seems, on subsequent calls asused_default_charcan be non-zero prior to the call. A non-zero value there, causes the tzname to be set to the empty string even if the conversion succeeded.zooba commentedon May 20, 2019
I've received a detailed response from the UCRT team, and there are a few pieces here.
Perhaps it would be easy to do the replacement of underscores with hyphens on Windows in this function? I think that's safe enough, yes?
CharlieClark commentedon May 20, 2019
I can confirm that using "de-DE" does indeed avoid the crash.
jeremykloth commentedon May 21, 2019
Even some well known locale names still use the utf-8 code page. Most seem
to uncommon, but at least es-BR (Brazil) does and would still fall victim
to these UCRT bugs.
eryksun commentedon Feb 6, 2020
Recent releases of ucrt implement this translation from underscore to hyphen for us, so this suggestion is no longer always necessary. However, I don't know what the PEP-11 stance is regarding the modern lifecycle policy of Windows 10. Will Python support a release for as long as it's supported by the enterprise version? For example, support for 1709 enterprise ends on 2020-04-14, so if we followed that, then Python 3.9 would require Windows 10 1803 or higher. That seems wrong while we're still supporting Windows 8.1, but what will the stance be when Python supports only Windows versions that use the modern lifecycle?
eryksun commentedon Feb 6, 2020
es-BR is a custom locale for the Spanish language in Brazil, as opposed to the common Portuguese locale (pt-BR). It's a Unicode-only locale, which means its ANSI codepage is 0. Since 0 is CP_ACP, its effective ANSI codepage is the system or process ANSI codepage.
For example:
Portuguese in Brazil uses codepage 1252 as its ANSI codepage:
>>> n = kernel32.GetLocaleInfoEx('pt-BR', 0x1004, buf, 10) >>> buf.value '1252'Spanish in Brazil uses CP_ACP:
>>> n = kernel32.GetLocaleInfoEx('es-BR', 0x1004, buf, 10) >>> buf.value '0'hi-IN (Hindi, India) is a common Unicode-only locale:
>>> n = kernel32.GetLocaleInfoEx('hi-IN', 0x1004, buf, 10) >>> buf.value '0'ucrt has switched to using UTF-8 for Unicode-only locales:
>>> locale.setlocale(locale.LC_CTYPE, 'hi_IN') 'hi_IN' >>> ucrt = ctypes.CDLL('ucrtbase', use_errno=True) >>> ucrt.___lc_codepage_func() 65001Note that ucrt uses UTF-8 for Unicode-only locales only when using an explicitly named locale such as "hi_IN", "Hindi_India" or even just "Hindi". On the other hand, if a Unicode-only locale is used implicitly, ucrt instead uses the system ANSI codepage:
>>> locale.setlocale(locale.LC_CTYPE, '') 'Hindi_India.1252' >>> ucrt.___lc_codepage_func() 1252I suppose this is for backwards compatibility. Windows 10 at least supports setting the system ANSI codepage to UTF-8, or overriding the process ANSI codepage to UTF-8 via the application manifest "actveCodePage" setting. For the latter, I modified the manifest in a "python_utf8.exe" copy of the normal "python.exe" binary, which is simpler than having to reboot to change the system locale:
>>> import locale >>> locale.setlocale(locale.LC_CTYPE, '') 'Hindi_India.utf8'zooba commentedon Feb 12, 2020
This needs a separate discussion on python-dev, but for issues like this I think it's fine to blame the underlying platform and recommend getting their updates. We don't have to work around every Windows bug for all time.
I'll read the rest of what's been happening on this issue when I'm less tired, but not quite up to it right now ;)
eryksun commentedon Mar 27, 2021
ucrt in Windows 10 v2004 uses an internal wide-character version of the time-zone name, which gets updated by _tzset() and kept in sync with the _tzname encoded strings. Also, note that the current implementation uses the locale's ANSI code page for both "de-DE" and "de_DE". UTF-8 has to be requested explicitly when using a BCP-47 locale name, such as "de_DE.UTF-8" or "de-DE.UTF-8", and UTF-8 is the only explicit codeset that's allowed in this case.