test_ssl and test_httplib.HTTPSTest crash randomly with "Windows fatal exception: access violation" on Windows

@db3l

BPO	44252
Nosy	@db3l, @vstinner, @tiran, @pablogsal, @miss-islington, @erlend-aasland, @Fidget-Spinner
PRs	bpo-44252: Correctly implement gc support for SSLError objects #26439 [3.10] bpo-44252: Correctly implement gc support for SSLError objects (GH-26439) #26441

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2021-05-29.00:11:19.695>
created_at = <Date 2021-05-27.17:13:50.782>
labels = ['expert-SSL', 'tests', '3.11']
title = 'test_ssl and test_httplib.HTTPSTest crash randomly with "Windows fatal exception: access violation" on Windows'
updated_at = <Date 2021-05-29.00:11:19.694>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-05-29.00:11:19.694>
actor = 'pablogsal'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2021-05-29.00:11:19.695>
closer = 'pablogsal'
components = ['Tests', 'SSL']
creation = <Date 2021-05-27.17:13:50.782>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 44252
keywords = ['patch']
message_count = 28.0
messages = ['394573', '394581', '394599', '394600', '394605', '394606', '394608', '394614', '394615', '394617', '394619', '394621', '394624', '394654', '394655', '394661', '394685', '394687', '394688', '394691', '394693', '394694', '394695', '394696', '394698', '394699', '394703', '394704']
nosy_count = 7.0
nosy_names = ['db3l', 'vstinner', 'christian.heimes', 'pablogsal', 'miss-islington', 'erlendaasland', 'kj']
pr_nums = ['26439', '26441']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44252'
versions = ['Python 3.11']

Christian Heimes reported a crash in bpo-42972:
https://bugs.python.org/issue42972#msg394520
---
#70587 is failing with an access violation on Windows. It's failing in one of the flaky tests. I wonder if the segfault is related to flaky tests somehow...

https://dev.azure.com/Python/cpython/_build/results?buildId=81570&view=logs&j=c83831cd-3752-5cc7-2f01-8276919eb334

test_pha_optional (test.test_ssl.TestPostHandshakeAuth) ... ok
test_pha_optional_nocert (test.test_ssl.TestPostHandshakeAuth) ... ok
test_pha_required (test.test_ssl.TestPostHandshakeAuth) ... ok
Windows fatal exception: access violation

Current thread 0x000009e0 (most recent call first):
File "D:\a\1\s\lib\linecache.py", line 63 in checkcache
File "D:\a\1\s\lib\traceback.py", line 375 in extract
File "D:\a\1\s\lib\traceback.py", line 494 in __init__
File "D:\a\1\s\lib\traceback.py", line 132 in format_exception
File "D:\a\1\s\lib\test\test_ssl.py", line 262 in handle_error
File "D:\a\1\s\lib\test\test_ssl.py", line 2530 in run
File "D:\a\1\s\lib\threading.py", line 1006 in _bootstrap_inner
File "D:\a\1\s\lib\threading.py", line 963 in _bootstrap

Thread 0x000003c4 (most recent call first):
File "D:\a\1\s\lib\threading.py", line 1102 in _wait_for_tstate_lock
File "D:\a\1\s\lib\threading.py", line 1086 in join
File "D:\a\1\s\lib\test\test_ssl.py", line 2604 in run
File "D:\a\1\s\lib\threading.py", line 1006 in _bootstrap_inner
File "D:\a\1\s\lib\threading.py", line 963 in _bootstrap

Thread 0x00001700 (most recent call first):
File "D:\a\1\s\lib\ssl.py", line 1131 in read
File "D:\a\1\s\lib\ssl.py", line 1256 in recv
File "D:\a\1\s\lib\test\test_ssl.py", line 4471 in test_pha_required_nocert
File "D:\a\1\s\lib\unittest\case.py", line 549 in _callTestMethod
File "D:\a\1\s\lib\unittest\case.py", line 592 in run
File "D:\a\1\s\lib\unittest\case.py", line 652 in __call__
File "D:\a\1\s\lib\unittest\suite.py", line 122 in run
File "D:\a\1\s\lib\unittest\suite.py", line 84 in __call__
File "D:\a\1\s\lib\unittest\suite.py", line 122 in run
File "D:\a\1\s\lib\unittest\suite.py", line 84 in __call__
File "D:\a\1\s\lib\unittest\runner.py", line 176 in run
File "D:\a\1\s\lib\test\support\init.py", line 959 in _run_suite
File "D:\a\1\s\lib\test\support\init.py", line 1082 in run_unittest
File "D:\a\1\s\lib\test\test_ssl.py", line 5007 in test_main
File "D:\a\1\s\lib\test\libregrtest\runtest.py", line 246 in _runtest_inner2
File "D:\a\1\s\lib\test\libregrtest\runtest.py", line 282 in _runtest_inner
File "D:\a\1\s\lib\test\libregrtest\runtest.py", line 154 in _runtest
File "D:\a\1\s\lib\test\main.py", line 2 in <module>
File "D:\a\1\s\lib\runpy.py", line 86 in _run_code
File "D:\a\1\s\lib\runpy.py", line 196 in _run_module_as_main
##[error]Cmd.exe exited with code '-1073741819'.
---

Erlend E. Aasland added:
---
Hm, I'm unable to reproduce it w/addr sanitiser on macOS (FWIW).

$ ./python.exe -m test test_ssl -F -u all -m test_pha_required_nocert

Passing 1000 successful runs now. I'll see if I can get a Win dev env set up later.
---

I may be related to commit dcb8786 "bpo-42972: Fully implement GC protocol for ssl heap types (GH-26370)".

GH 26411, run id 2687367477 also failed with an access violation:
https://github.com/python/cpython/pull/26411/checks?check_run_id=2687367477

Not in test_pha_required_nocert, but test_local_bad_hostname.

May be related.

Crash also seen on test_httplib:

(...)
test_attributes (test.test_httplib.HTTPSTest) ... ok
test_host_port (test.test_httplib.HTTPSTest) ... ok
Windows fatal exception: access violation

Current thread 0x000011ec (most recent call first):
File "D:\a\cpython\cpython\lib\socket.py", line 502 in close
File "D:\a\cpython\cpython\lib\ssl.py", line 1072 in _create
File "D:\a\cpython\cpython\lib\ssl.py", line 518 in wrap_socket
File "D:\a\cpython\cpython\lib\test\ssl_servers.py", line 37 in get_request
File "D:\a\cpython\cpython\lib\socketserver.py", line 311 in _handle_request_noblock
File "D:\a\cpython\cpython\lib\socketserver.py", line 237 in serve_forever
File "D:\a\cpython\cpython\lib\test\ssl_servers.py", line 142 in run
File "D:\a\cpython\cpython\lib\threading.py", line 1006 in _bootstrap_inner
File "D:\a\cpython\cpython\lib\threading.py", line 963 in _bootstrap

Thread 0x000013ec (most recent call first):
File "D:\a\cpython\cpython\lib\socket.py", line 496 in _real_close
File "D:\a\cpython\cpython\lib\ssl.py", line 1330 in _real_close
File "D:\a\cpython\cpython\lib\socket.py", line 502 in close
File "D:\a\cpython\cpython\lib\ssl.py", line 1072 in _create
File "D:\a\cpython\cpython\lib\ssl.py", line 518 in wrap_socket
File "D:\a\cpython\cpython\lib\http\client.py", line 1448 in connect
File "D:\a\cpython\cpython\lib\http\client.py", line 969 in send
File "D:\a\cpython\cpython\lib\http\client.py", line 1031 in _send_output
File "D:\a\cpython\cpython\lib\http\client.py", line 1271 in endheaders
File "D:\a\cpython\cpython\lib\http\client.py", line 1322 in _send_request
File "D:\a\cpython\cpython\lib\http\client.py", line 1276 in request
File "D:\a\cpython\cpython\lib\test\test_httplib.py", line 1950 in test_local_bad_hostname
File "D:\a\cpython\cpython\lib\unittest\case.py", line 549 in _callTestMethod
File "D:\a\cpython\cpython\lib\unittest\case.py", line 592 in run
File "D:\a\cpython\cpython\lib\unittest\case.py", line 652 in __call__
File "D:\a\cpython\cpython\lib\unittest\suite.py", line 122 in run
File "D:\a\cpython\cpython\lib\unittest\suite.py", line 84 in __call__
File "D:\a\cpython\cpython\lib\unittest\suite.py", line 122 in run
File "D:\a\cpython\cpython\lib\unittest\suite.py", line 84 in __call__
File "D:\a\cpython\cpython\lib\unittest\suite.py", line 122 in run
File "D:\a\cpython\cpython\lib\unittest\suite.py", line 84 in __call__
File "D:\a\cpython\cpython\lib\unittest\runner.py", line 176 in run
File "D:\a\cpython\cpython\lib\test\support\init.py", line 959 in _run_suite
File "D:\a\cpython\cpython\lib\test\support\init.py", line 1082 in run_unittest
File "D:\a\cpython\cpython\lib\test\libregrtest\runtest.py", line 210 in _test_module
File "D:\a\cpython\cpython\lib\test\libregrtest\runtest.py", line 246 in _runtest_inner2
File "D:\a\cpython\cpython\lib\test\libregrtest\runtest.py", line 282 in _runtest_inner
File "D:\a\cpython\cpython\lib\test\libregrtest\runtest.py", line 154 in _runtest
File "D:\a\cpython\cpython\lib\test\libregrtest\runtest.py", line 194 in runtest
File "D:\a\cpython\cpython\lib\test\libregrtest\main.py", line 321 in rerun_failed_tests
File "D:\a\cpython\cpython\lib\test\libregrtest\main.py", line 698 in _main
File "D:\a\cpython\cpython\lib\test\libregrtest\main.py", line 641 in main
File "D:\a\cpython\cpython\lib\test\libregrtest\main.py", line 719 in main
File "D:\a\cpython\cpython\lib\test\main.py", line 2 in <module>
File "D:\a\cpython\cpython\lib\runpy.py", line 86 in _run_code
File "D:\a\cpython\cpython\lib\runpy.py", line 196 in _run_module_as_main
test_local_bad_hostname (test.test_httplib.HTTPSTest) ...
Error: Process completed with exit code 1.

test.pythoninfo:

ssl.HAS_SNI: True
ssl.OPENSSL_VERSION: OpenSSL 1.1.1k 25 Mar 2021
ssl.OPENSSL_VERSION_INFO: (1, 1, 1, 11, 15)
ssl.OP_ALL: -0x7fffffac
ssl.OP_NO_TLSv1_1: 0x10000000
ssl.SSLContext.maximum_version: MAXIMUM_SUPPORTED
ssl.SSLContext.minimum_version: TLSv1_2
ssl.SSLContext.options: OP_NO_COMPRESSION|OP_ENABLE_MIDDLEBOX_COMPAT|OP_CIPHER_SERVER_PREFERENCE|OP_NO_SSLv3|0x54
ssl.SSLContext.protocol: PROTOCOL_TLS_CLIENT
ssl.SSLContext.verify_mode: CERT_REQUIRED
ssl.default_https_context.maximum_version: MAXIMUM_SUPPORTED
ssl.default_https_context.minimum_version: TLSv1_2
ssl.default_https_context.options: OP_NO_COMPRESSION|OP_ENABLE_MIDDLEBOX_COMPAT|OP_CIPHER_SERVER_PREFERENCE|OP_NO_SSLv3|0x54
ssl.default_https_context.protocol: PROTOCOL_TLS_CLIENT
ssl.default_https_context.verify_mode: CERT_REQUIRED
ssl.stdlib_context.maximum_version: MAXIMUM_SUPPORTED
ssl.stdlib_context.minimum_version: TLSv1_2
ssl.stdlib_context.options: OP_NO_COMPRESSION|OP_ENABLE_MIDDLEBOX_COMPAT|OP_CIPHER_SERVER_PREFERENCE|OP_NO_SSLv3|0x54
ssl.stdlib_context.protocol: PROTOCOL_TLS_CLIENT
ssl.stdlib_context.verify_mode: CERT_NONE

I forgot the link:
https://github.com/python/cpython/pull/26411/checks?check_run_id=2687367477

Oh right, Erlend already reported exactly the same crash :-)

Is this happening on the current 3.10 or just with a specific PR?

Maybe the problem is related to Python frames. I think Mark Shannon made changes related to frames recently in 3.10 and main branches.

It's strange that I got such error:

File "C:\vstinner\python\main\lib\unittest\result.py", line 205, in _is_relevant_tb_level
return '__unittest' in tb.tb_frame.f_globals
TypeError: argument of type 'NoneType' is not iterable

It's also strange that two crashes occurred while reading a traceback:

Windows fatal exception: access violation

Current thread 0x000009e0 (most recent call first):
File "D:\a\1\s\lib\linecache.py", line 63 in checkcache
File "D:\a\1\s\lib\traceback.py", line 375 in extract
File "D:\a\1\s\lib\traceback.py", line 494 in __init__
File "D:\a\1\s\lib\traceback.py", line 132 in format_exception
File "D:\a\1\s\lib\test\test_ssl.py", line 262 in handle_error

and:

Fatal Python error: Segmentation fault

Current thread 0x00007f9de2d1f640 (most recent call first):
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/linecache.py", line 72 in checkcache
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/traceback.py", line 375 in extract
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/traceback.py", line 494 in __init__
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/traceback.py", line 132 in format_exception
File "/home/buildbot/buildarea/3.10.cstratak-fedora-stable-x86_64/build/Lib/test/test_ssl.py", line 262 in handle_error

Extract of test_ssl.py:

def handle_error(prefix):
    exc_format = ' '.join(traceback.format_exception(*sys.exc_info()))
    ...

Maybe sometimes somehow, frames are invalid.

Maybe test_ssl/test_httplib only makes the crash more likely, but there is a regression to frames somewhere.

Oh, I forgot to mention that I saw the "TypeError: argument of type 'NoneType' is not iterable" error on the main branch, not on the 3.10 branch.

I remove Mark from the issue. I may open later a different issue for tb_frame.f_globals=None, but it's unrelated to this bug.

I can reproduce the issue with a small hack to make GC collections make likely:

diff --git a/Lib/site.py b/Lib/site.py
index 939893eb5e..4103792e84 100644
--- a/Lib/site.py
+++ b/Lib/site.py
@@ -672,3 +672,6 @@ def exists(path):
 
 if __name__ == '__main__':
     _script()
+
+import gc
+gc.set_threshold(5)

With this patch, it becomes trivial to reproduce the crash on Linux:

$ ./python -m test test_ssl -m test_ssl_cert_verify_error -v 
(...)
Fatal Python error: Segmentation fault

Current thread 0x00007fd0f72a8640 (most recent call first):
File "/home/vstinner/python/3.10/Lib/traceback.py", line 132 in format_exception
File "/home/vstinner/python/3.10/Lib/test/test_ssl.py", line 262 in handle_error
File "/home/vstinner/python/3.10/Lib/test/test_ssl.py", line 2401 in wrap_conn
(...)

So the problem is that _ssl.SSLError type traverse function is NULL.

For me, there are different issues:

PyType_FromSpec() must fail if a type is declared with Py_TPFLAGS_HAVE_GC but its tp_traverse function is NULL.
_ssl.SSLError must implement the tp_traverse slot

Moreover, in debug mode, PyObject_GC_Track() calls the traverse function to ensure that it's safe to call it. PyObject_GC_Track() must crash of tp_traverse is NULL. If it didn't track, it means that the _ssl.SSLError instance was not tracked by the GC. Is that ok? Objects implementing the GC protocol should be tracked by the GC, no?

An exception is *likely* to be part of a reference cycle because it contains a traceback which contains frames which contains variables, and one variable can be the exception.

The regression was introduced by

dcb8786

Oddly, it turns out it's the "--junit-xml" parameter during buildbot tests that is making it reproducible there.

Pass: python_d -m test.regrtest test_ssl
Fail: python_d -m test.regrtest --junit-xml out.xml test_ssl

The latter consistently fails all of the time for me manually.

I also found what I think is the same commit Pablo just referenced (albeit the cherry pick version on the 3.10 branch) - ea47a8a. Reverting just that commit as part of the latest 3.10 source passes.

Easier reproducer:

>>> import gc
>>> gc.get_referrers(x)
AddressSanitizer:DEADLYSIGNAL

=================================================================
==17017==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000000000 bp 0x7fff93f8f7a0 sp 0x7fff93f8f758 T0)
==17017==Hint: pc points to the zero page.
==17017==The signal is caused by a READ memory access.
==17017==Hint: address points to the zero page.
#0 0x0 (<unknown module>)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (<unknown module>)
==17017==ABORTING

New changeset 8b4312b by Pablo Galindo in branch 'main':
bpo-44252: Correctly implement gc support for SSLError objects (GH-26439)
8b4312b

New changeset a7aa105 by Miss Islington (bot) in branch '3.10':
bpo-44252: Correctly implement gc support for SSLError objects (GH-26439) (GH-26441)
a7aa105

Navigation Menu

Repository navigation

test_ssl and test_httplib.HTTPSTest crash randomly with "Windows fatal exception: access violation" on Windows #88418

14 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test_ssl and test_httplib.HTTPSTest crash randomly with "Windows fatal exception: access violation" on Windows #88418

Description

Activity

vstinner commented on May 27, 2021

erlend-aasland commented on May 27, 2021

vstinner commented on May 27, 2021

vstinner commented on May 27, 2021

pablogsal commented on May 27, 2021

14 remaining items

vstinner commented on May 28, 2021

vstinner commented on May 28, 2021

vstinner commented on May 28, 2021

vstinner commented on May 28, 2021

pablogsal commented on May 28, 2021

db3l commented on May 28, 2021

pablogsal commented on May 28, 2021

pablogsal commented on May 28, 2021

pablogsal commented on May 28, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions