Repository navigation
[Windows] time: crash on formatting time with de_DE locale #80973
Description
Activity
CharlieClark commented
on May 4, 2019 CharlieClarkmannequinMannequinAuthorBased on a bug report (https://bitbucket.org/openpyxl/openpyxl/issues/1266/locale) from a user of the openpyxl library I've identified a bug in the zipfile module that causes the Python process to crash on Windows. Currently tested with Python 3.7.3 (32-bit on Windows 10).
Sample code
import faulthandler import locale from zipfile import ZipFile faulthandler.enable() locale.setlocale(locale.LC_ALL, 'de_DE') out = open("out.zip", "wb") archive = ZipFile(out, "w") archive.writestr("properties.xml", b"<workbookPr/>")
faulthandler fingers line 1757 as the culprit but running this line locally does not cause the crash. The issue seems to be limited to Windows.
Zheaoli commented
on May 5, 2019 ZheaolimannequinMannequinI can't reproduce this error on my system by using the same code.
Could you share the system info with us?
Zheaoli commented
on May 5, 2019 ZheaolimannequinMannequinOr would you share the Exception with us ?
I guess it's caused by system setting
CharlieClark commented
on May 5, 2019 CharlieClarkmannequinMannequinAuthorPython 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] on win32
Zheaoli commented
on May 5, 2019 ZheaolimannequinMannequinSame environment.
But I still can not reproduce this exception. I guess maybe it's about the local time or timezone problem. I will find a way to figure it out
CharlieClark commented
on May 5, 2019 CharlieClarkmannequinMannequinAuthorThat's what we thought when we looked at it, but as I said, I couldn't reproduce it with just the
timecall or theZInfoinstantiation, so something odd is happening. I do have a German version of Windows as I suspect the original reporter does. You'd think that sort of thing wouldn't matter…DominikGeldmacher commented
on May 5, 2019 DominikGeldmachermannequinMannequinI can reproduce it on Python 3.7.3 german Windows10 enterprise
Windows fatal exception: code 0xc0000374
Current thread 0x00003bc8 (most recent call first):
File "C:\Python37\lib\zipfile.py", line 1757 in writestrZheaoli commented
on May 5, 2019 ZheaolimannequinMannequinHi Dominik Geldmacher
May I get your system version? 1809 or 1903?
I guess maybe it's microsoft error
DominikGeldmacher commented
on May 5, 2019 DominikGeldmachermannequinMannequinEnterprise 1809 and Professional 1803 (german localized) both reproduce the issue
Zheaoli commented
on May 5, 2019 ZheaolimannequinMannequincopy that
I will reset my locale setting to figure it out
CharlieClark commented
on May 5, 2019 CharlieClarkmannequinMannequinAuthorwinver tells me I have 1809. I'm only using Windows in a VM so I'm not that familiar with its innards.
Also get the error with WinPython 3.6:
Windows fatal exception: code 0xc0000374
Current thread 0x000010c0 (most recent call first):
File "C:\Users\charlieclark\WPy64-3680\python-3.6.8.amd64\lib\zipfile.py", line 1658 in writestr
File "<stdin>", line 1 in <module>16 remaining items
CharlieClark commented
on May 7, 2019 CharlieClarkmannequinMannequinAuthorAnd this is the result.
old locale: C
count: 28
value: Mitteleuropäische Sommerzeit
new locale: de_DE
count: -1
value:
Windows fatal exception: code 0xc0000374Looks like
print('new locale:', crt_locale._wsetlocale(0, 'de_DE')) print('count:', crt_time.wcsftime(wbuf, 1024, '%Z', tm))
is the culprit.
Final test, this time, no Python what so ever. I've added a zip containing a simple C program (source and .exe) that performs the same test.
The output should be similar to:
--------
The current locale is now: C
The time zone is: 'Mountain Daylight Time' (22 characters)
--------
The updated locale is now: de_DE
The time zone is: 'Mountain Daylight Time' (22 characters)
--------If this crashes, the problem is then within the CRT itself :(
CharlieClark commented
on May 7, 2019 CharlieClarkmannequinMannequinAuthorThis is the result
\bpo-36792>test.exe
--------
The current locale is now: C
The time zone is: 'Mitteleuropõische Sommerzeit' (28 characters)
--------
The updated locale is now: de_DE
The time zone is: '' (-1 characters)
--------
NB something is wrong with that string. Should be Mitteleuropäsiche Sommerzeit but that could just be an encoding thing.Thanks again! I will have some more tests for you to try tomorrow as I am out of time for today.
I'm currently of the belief that there is something Python is going to have to do to work around an issue within the CRT, but more testing will prove that theory.
I have managed to setup a VM that can reproduce the error. Unfortunately, the error (heap corruption) is coming from within the UCRT. Attempting to work around that, I came across another error in the UCRT.
Due to these errors in all available UCRT versions, the only available solution is to re-implement %Z handling in Python's implementation of strftime().
@steve.dower: If you could forward the following information to the interested parties for faster response, it would be great.
Everyone else, what follows contains UCRT internal details which is of no interest to Python.
In wcsftime() [ucrt\time\wcsftime.cpp, line 1018], the call to _mbstowcs_s_l() doesn't handle the error case of EILSEQ. This error happens when the tzname (originally set by __tzset() from the Unicode name in the registry encoded to the ACP) contains an undecodable character. In the 'de_DE' case, \u00E4. _mbstowcs_s_l returns without changing
wnum(which is 0). The 'Z' case then blindly subtracts 1 (wnum=-1) and advances the string pointer, now pointing before the start when the format is just L"%Z" (like Python has). Now when wcsftime() is returning it stores a '\0' at that current position thus causing a heap corruption.Ok, no deal breaker, we can just update the _tzname array when we call setlocale(). Alas, another error :( If _tzset() is called a second time, it fails when encoding the Unicode name due to an uninitialized variable. In tzset_from_system_nolock() [ucrt\time\tzset.cpp, line 158], the function __acrt_WideCharToMultiByte() [appcrt\locale\widechartomultibyte.cpp] does not set the
lpUsedDefaultCharflag when FALSE. This is only an issue, it seems, on subsequent calls asused_default_charcan be non-zero prior to the call. A non-zero value there, causes the tzname to be set to the empty string even if the conversion succeeded.I've received a detailed response from the UCRT team, and there are a few pieces here.
- the fact that tzname is cached in ACP is known and will be fixed
- the decoding bug is real, but it's due to the experimental UTF-8 support
- the experimental UTF-8 support was enabled because "de_DE" is not a known Windows locale name - try with "de-DE"
Perhaps it would be easy to do the replacement of underscores with hyphens on Windows in this function? I think that's safe enough, yes?
CharlieClark commented
on May 20, 2019 CharlieClarkmannequinMannequinAuthorI can confirm that using "de-DE" does indeed avoid the crash.
jeremykloth commented
on May 21, 2019 jeremyklothmannequinMannequin- the experimental UTF-8 support was enabled because "de_DE" is not a
known Windows locale name - try with "de-DE"
Perhaps it would be easy to do the replacement of underscores with hyphens
on Windows in this function? I think that's safe enough, yes?Even some well known locale names still use the utf-8 code page. Most seem
to uncommon, but at least es-BR (Brazil) does and would still fall victim
to these UCRT bugs.- the experimental UTF-8 support was enabled because "de_DE" is not a
Perhaps it would be easy to do the replacement of underscores with
hyphens on Windows in this function? I think that's safe enough, yes?Recent releases of ucrt implement this translation from underscore to hyphen for us, so this suggestion is no longer always necessary. However, I don't know what the PEP-11 stance is regarding the modern lifecycle policy of Windows 10. Will Python support a release for as long as it's supported by the enterprise version? For example, support for 1709 enterprise ends on 2020-04-14, so if we followed that, then Python 3.9 would require Windows 10 1803 or higher. That seems wrong while we're still supporting Windows 8.1, but what will the stance be when Python supports only Windows versions that use the modern lifecycle?
Even some well known locale names still use the utf-8 code page. Most
seem to uncommon, but at least es-BR (Brazil) does and would still
fall victim to these UCRT bugs.es-BR is a custom locale for the Spanish language in Brazil, as opposed to the common Portuguese locale (pt-BR). It's a Unicode-only locale, which means its ANSI codepage is 0. Since 0 is CP_ACP, its effective ANSI codepage is the system or process ANSI codepage.
For example:
>>> kernel32 = ctypes.WinDLL('kernel32', use_last_error=True) >>> buf = (ctypes.c_wchar * 10)()Portuguese in Brazil uses codepage 1252 as its ANSI codepage:
>>> n = kernel32.GetLocaleInfoEx('pt-BR', 0x1004, buf, 10) >>> buf.value '1252'Spanish in Brazil uses CP_ACP:
>>> n = kernel32.GetLocaleInfoEx('es-BR', 0x1004, buf, 10) >>> buf.value '0'hi-IN (Hindi, India) is a common Unicode-only locale:
>>> n = kernel32.GetLocaleInfoEx('hi-IN', 0x1004, buf, 10) >>> buf.value '0'ucrt has switched to using UTF-8 for Unicode-only locales:
>>> locale.setlocale(locale.LC_CTYPE, 'hi_IN') 'hi_IN' >>> ucrt = ctypes.CDLL('ucrtbase', use_errno=True) >>> ucrt.___lc_codepage_func() 65001Note that ucrt uses UTF-8 for Unicode-only locales only when using an explicitly named locale such as "hi_IN", "Hindi_India" or even just "Hindi". On the other hand, if a Unicode-only locale is used implicitly, ucrt instead uses the system ANSI codepage:
>>> locale.setlocale(locale.LC_CTYPE, '') 'Hindi_India.1252' >>> ucrt.___lc_codepage_func() 1252I suppose this is for backwards compatibility. Windows 10 at least supports setting the system ANSI codepage to UTF-8, or overriding the process ANSI codepage to UTF-8 via the application manifest "actveCodePage" setting. For the latter, I modified the manifest in a "python_utf8.exe" copy of the normal "python.exe" binary, which is simpler than having to reboot to change the system locale:
C:\>python_utf8 -q>>> import locale >>> locale.setlocale(locale.LC_CTYPE, '') 'Hindi_India.utf8'Will Python support a release for as long as it's supported by the enterprise version? For example, support for 1709 enterprise ends on 2020-04-14, so if we followed that, then Python 3.9 would require Windows 10 1803 or higher. That seems wrong while we're still supporting Windows 8.1, but what will the stance be when Python supports only Windows versions that use the modern lifecycle?
This needs a separate discussion on python-dev, but for issues like this I think it's fine to blame the underlying platform and recommend getting their updates. We don't have to work around every Windows bug for all time.
I'll read the rest of what's been happening on this issue when I'm less tired, but not quite up to it right now ;)
ucrt in Windows 10 v2004 uses an internal wide-character version of the time-zone name, which gets updated by _tzset() and kept in sync with the _tzname encoded strings. Also, note that the current implementation uses the locale's ANSI code page for both "de-DE" and "de_DE". UTF-8 has to be requested explicitly when using a BCP-47 locale name, such as "de_DE.UTF-8" or "de-DE.UTF-8", and UTF-8 is the only explicit codeset that's allowed in this case.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: