You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test_site.StartupImportTests.test_startup_imports fails if default code page is cp65001 #80959
Windows desktop skus have a default ANSI codepage (returned by GetACP()) of 1252 (Western European). Windows IoT Core and Windows Nano Server have a default codepage of 65001 (UTF-8).
This causes test_site.StartupImportTests.test_startup_imports to fail on Windows IoT Core and Windows Nano Server because cp65001.py is loaded instead of the frozen cp1252.py at startup.
I tried changing the default codepage to 65001 on my dev machine and rebuilding Python and it had no effect that I could tell on the generated frozen importlibs.
The simplest solutions would be for the test_startup_imports test to be skipped or changed to pass when the locale.getpreferredencoding() returns 'cp65001'
PyUnicode_AsUnicodeAndSize is neutral about storing surrogate codes in a 16-bit wchar_t string. In particular, the Python string in this case contains two surrogate codes, but they're passed to WideCharToMultiByte as a UTF-16 surrogate pair for the single character U+10000.
Anyway, it seems to me this issue will be resolved if cp65001.py is rewritten without functools.partial.
I think it is better to just make the check in the test conditional.
Okay. The test verifies work done to minimize interpreter startup time, but probably the relative cost of importing functools (and thus collections et al) isn't significant compared to the overall cost of spawning a process in a Windows desktop environment. That may not be the case for Nano Server and IoT Core.
changed the title [-]test_site.StartupImportTests.test_startup_imports fails if default code page is not cp1252[/-][+]test_site.StartupImportTests.test_startup_imports fails if default code page is cp65001[/+]on May 6, 2019
So the suggested change makes the locale module internally consistent on Windows and more transparent for anyone who doesn't know off the top of their head that "cp65001" is just UTF-8.
paulmon commented on May 10, 2019
paulmonmannequin
MannequinAuthor
More actions
I can verify that PR 13110 fixes the issue with test_startup_imports on Windows IoT Core ARM32
paulmon commented on May 10, 2019
paulmonmannequin
MannequinAuthor
More actions
Sorry that was supposed to say:
I can verify that PR 13230 fixes the issue with test_startup_imports on Windows IoT Core ARM32
So I don't think it is lie. It is just "what encoding name we should choose when GetACP() returned 65001.".
With your PR 13230, cp65001 is truly utf-8. So returning "utf-8" seems right behavior.
About the ANSI code page, Lib/encodings/init.py calls _winapi.GetACP() to avoid relying on locale.getpreferredencoding() which lies when UTF-8 Mode is enabled:
So I don't think it is lie. It is just "what encoding name we should choose when GetACP() returned 65001.".
With your PR 13230, cp65001 is truly utf-8. So returning "utf-8" seems right behavior.
Well, feel free to propose a PR. I have no strong opinion on this level of detail :-)
Paul Monson: Your initial issue has been fixed in the master branch.
I'm not sure what are Windows IoT Core and Windows Nano Server. Do you care of Python 3.7? If someone wants to support running test_site with ANSI code page set to 65001, I suggest to fix test_site directly like PR 13072 in Python 3.7. My attempt to avoid functools made cp65001 codec way slower. Fixing one specific test should not make Python that much slower ;-)
paulmon commented on May 10, 2019
paulmonmannequin
MannequinAuthor
More actions
Thanks Victor! Since we aren't backporting ARM32 changes, I don't think it's important to fix this test in 3.7. I am trying to get the buildbot tests for Windows ARM32 to zero errors.
Since we aren't backporting ARM32 changes, I don't think it's important to fix this test in 3.7. I am trying to get the buildbot tests for Windows ARM32 to zero errors.
Activity
paulmon commentedon May 2, 2019
Windows desktop skus have a default ANSI codepage (returned by GetACP()) of 1252 (Western European). Windows IoT Core and Windows Nano Server have a default codepage of 65001 (UTF-8).
This causes test_site.StartupImportTests.test_startup_imports to fail on Windows IoT Core and Windows Nano Server because cp65001.py is loaded instead of the frozen cp1252.py at startup.
I tried changing the default codepage to 65001 on my dev machine and rebuilding Python and it had no effect that I could tell on the generated frozen importlibs.
The simplest solutions would be for the test_startup_imports test to be skipped or changed to pass when the locale.getpreferredencoding() returns 'cp65001'
methane commentedon May 3, 2019
Could you paste how the test fails?
paulmon commentedon May 3, 2019
======================================================================
FAIL: test_startup_imports (test.test_site.StartupImportTests)
----------------------------------------------------------------------
methane commentedon May 3, 2019
@victor It seems you added cp65001 as Windows-only encoding in bpo-13216.
How do you think about removing cp65001 encoding, and add 'cp65001' -> 'utf_8' alias which is available on all platforms?
vstinner commentedon May 4, 2019
cp65001 is *not* utf-8: Microsoft decided to handle surrogates differently
for some reasons.
eryksun commentedon May 4, 2019
Do you mean valid UTF-16 surrogate pairs? For example:
>>> codecs.code_page_encode(65001, '\ud800\udc00') (b'\xf0\x90\x80\x80', 2)PyUnicode_AsUnicodeAndSize is neutral about storing surrogate codes in a 16-bit wchar_t string. In particular, the Python string in this case contains two surrogate codes, but they're passed to WideCharToMultiByte as a UTF-16 surrogate pair for the single character U+10000.
Anyway, it seems to me this issue will be resolved if cp65001.py is rewritten without functools.partial.
serhiy-storchaka commentedon May 4, 2019
I think it is better to just make the check in the test conditional. It already contains some macOs specific conditions.
eryksun commentedon May 4, 2019
Okay. The test verifies work done to minimize interpreter startup time, but probably the relative cost of importing functools (and thus collections et al) isn't significant compared to the overall cost of spawning a process in a Windows desktop environment. That may not be the case for Nano Server and IoT Core.
vstinner commentedon May 6, 2019
Paul Monson: I'm unable to reproduce exactly your issue, but I tried to reproduce it partially using PYTHONIOENCODING=cp65001.
My PR 13110 avoids "import functools" at startup. Can you please try it and check if it fix test_site?
vstinner commentedon May 6, 2019
Victor:
Eryk:
Code page 65001 handles lone surrogate differently on Windows XP and older. It changed in Windows Vista:
https://unicodebook.readthedocs.io/operating_systems.html#encode-and-decode-functions
Steve Dower removed support for Vista from test_codecs.py 3 years ago:
commit f5aba58
Author: Steve Dower <steve.dower@microsoft.com>
Date: Tue Sep 6 19:42:27 2016 -0700
Maybe it's time to remove Lib/encodings/cp65001.py and add an alias cp65001 => utf_8 in Lib/encodings/aliases.py? See bpo-32592.
[-]test_site.StartupImportTests.test_startup_imports fails if default code page is not cp1252[/-][+]test_site.StartupImportTests.test_startup_imports fails if default code page is cp65001[/+]8 remaining items
vstinner commentedon May 9, 2019
My PR 13110 (avoid functools) makes codecs.lookup('cp65001').encode() made 2.7x slower:
#13110 (comment)
417 ns +- 17 ns
My PR 13230 (remove cp65001.py) makes it 1.5x faster :-)
#13230 (comment)
105 ns +- 3 ns
The reference is: 156 ns +- 3 ns.
vstinner commentedon May 9, 2019
I dislike lying in the locale module. This change is basically useless with my PR 13230.
eryksun commentedon May 9, 2019
Yes, functionally it's no different than using 'cp65001' as an alias. That said, the CRT special cases 65001 as "utf8":
>>> locale.setlocale(locale.LC_CTYPE, '') 'English_United Kingdom.utf8' >>> crt_locale = ctypes.CDLL('api-ms-win-crt-locale-l1-1-0', use_errno=True) >>> crt_locale.___lc_codepage_func() 65001So the suggested change makes the locale module internally consistent on Windows and more transparent for anyone who doesn't know off the top of their head that "cp65001" is just UTF-8.
paulmon commentedon May 10, 2019
I can verify that PR 13110 fixes the issue with test_startup_imports on Windows IoT Core ARM32
paulmon commentedon May 10, 2019
Sorry that was supposed to say:
I can verify that PR 13230 fixes the issue with test_startup_imports on Windows IoT Core ARM32
methane commentedon May 10, 2019
Note that Python produce "cpNNN" encoding name, not Windows.
cpython/Modules/_localemodule.c
Line 395 in 137be34
So I don't think it is lie. It is just "what encoding name we should choose when GetACP() returned 65001.".
With your PR 13230, cp65001 is truly utf-8. So returning "utf-8" seems right behavior.
vstinner commentedon May 10, 2019
New changeset d267ac2 by Victor Stinner in branch 'master':
bpo-36778: cp65001 encoding becomes an alias to utf_8 (GH-13230)
d267ac2
vstinner commentedon May 10, 2019
About the ANSI code page, Lib/encodings/init.py calls _winapi.GetACP() to avoid relying on locale.getpreferredencoding() which lies when UTF-8 Mode is enabled:
INADA-san:
Well, feel free to propose a PR. I have no strong opinion on this level of detail :-)
vstinner commentedon May 10, 2019
Paul Monson: Your initial issue has been fixed in the master branch.
I'm not sure what are Windows IoT Core and Windows Nano Server. Do you care of Python 3.7? If someone wants to support running test_site with ANSI code page set to 65001, I suggest to fix test_site directly like PR 13072 in Python 3.7. My attempt to avoid functools made cp65001 codec way slower. Fixing one specific test should not make Python that much slower ;-)
paulmon commentedon May 10, 2019
Thanks Victor! Since we aren't backporting ARM32 changes, I don't think it's important to fix this test in 3.7. I am trying to get the buildbot tests for Windows ARM32 to zero errors.
Windows IoT Core runs on Raspberry Pi and similar devices: https://developer.microsoft.com/en-us/windows/iot
Windows NanoServer is a very small version of Windows Server for running in Docker containers hosted on Windows Server.
vstinner commentedon May 10, 2019
Ok, thanks. I close the issue.
vstinner commentedon May 13, 2019
New changeset 3aef48e by Victor Stinner in branch 'master':
bpo-36778: Update cp65001 codec documentation (GH-13240)
3aef48e
vstinner commentedon Jun 4, 2019
New changeset ca612a9 by Victor Stinner in branch 'master':
bpo-36778: Remove outdated comment from CodePageTest (GH-13807)
ca612a9