Repository navigation
test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a) #80106
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-05-29.02:12:18.543>
created_at = <Date 2019-02-07.01:45:26.293>
labels = ['3.7', '3.8', 'type-bug', 'tests']
title = 'test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a)'
updated_at = <Date 2019-05-29.02:12:18.530>
user = 'https://github.com/pablogsal'bugs.python.org fields:
activity = <Date 2019-05-29.02:12:18.530>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-29.02:12:18.543>
closer = 'ned.deily'
components = ['Tests']
creation = <Date 2019-02-07.01:45:26.293>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 35925
keywords = ['patch']
message_count = 29.0
messages = ['334996', '335001', '335047', '335630', '335639', '335640', '335642', '335709', '335949', '335968', '336038', '336056', '336212', '336520', '336521', '340086', '340090', '341584', '341650', '341678', '341679', '341902', '341903', '341913', '342172', '342173', '342382', '342384', '343855']
nosy_count = 11.0
nosy_names = ['gregory.p.smith', 'vstinner', 'christian.heimes', 'benjamin.peterson', 'ned.deily', 'lukasz.langa', 'steve.dower', 'mdk', 'cstratak', 'pablogsal', 'miss-islington']
pr_nums = ['13124', '13139', '13252', '13253']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35925'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8']
Activity
pablogsal commentedon Feb 7, 2019
Example failures
https://buildbot.python.org/all/#/builders/117
https://buildbot.python.org/all/#/builders/106
======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------
Ran 105 tests in 2.477s
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
test_local_bad_hostname (test.test_httplib.HTTPSTest) ... server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
test_local_good_hostname (test.test_httplib.HTTPSTest) ... server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
Got an error:
[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1055)
test_local_unknown_cert (test.test_httplib.HTTPSTest) ... stopping HTTPS server
joining HTTPS thread
ok
Multiple SSL failures, also old commits that previously succeeded fail now. This seems something in the buildbot itself. Gregory, do you know if something SLL related was upgraded/modify in the gps-ubuntu-exynos5-armv7l worker?
gpshead commentedon Feb 7, 2019
FYI - the name of this bot is misleading. It is now Debian testing as of 18
hours ago instead of obsolete Ubuntu 14.04. I finally upgraded it.
Opens version says 1.1.1a.
--
blame half the typos on my phone.
On Wed, Feb 6, 2019, 5:45 PM Pablo Galindo Salgado <report@bugs.python.org
wrote:
> Traceback (most recent call last): > File > "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_httplib.py", > line 1629, in test_networked_good_cert > h.request('GET', '/') > File > "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", > line 1229, in request > self._send_request(method, url, body, headers, encode_chunked) > File > "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", > line 1275, in _send_request > self.endheaders(body, encode_chunked=encode_chunked) > File > "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", > line 1224, in endheaders > self._send_output(message_body, encode_chunked=encode_chunked) > File > "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", > line 1016, in _send_output > self.send(msg) > File > "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", > line 956, in send > self.connect() > File > "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", > line 1391, in connect > self.sock = self._context.wrap_socket(self.sock, > File > "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", > line 405, in wrap_socket > return self.sslsocket_class._create( > File > "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", > line 853, in _create > self.do_handshake() > File > "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", > line 1117, in do_handshake > self._sslobj.do_handshake() > ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate > verify failed: EE certificate key too weak (_ssl.c:1055) >gpshead commentedon Feb 7, 2019
I had emailed Christian around the same time you filed this.
"""
The problem likely not related to your hardware. I guess it's caused by
tightened crypto polices. OpenSSL 1.1.1 has disabled some weak crypto.
Some platforms like Debian and RHEL require even larger key sizes or
have disable some algorithms. Does the test also fail with the env var
OPENSSL_CONF set to a non-existing path?
""" - christian.heimes
testing that theory... setting OPENSSL_CONF=/invalid-path does indeed "fix" (work around) the failures. Presumably by relaxing the default system constraints.
I could have that env var set for this buildbot and eliminate the failure. But do we _want_ to do that? Anyone who compiles CPython and tries to run the test suite on a modern system with such an OpenSSL configuration is going to see similar failures and likely come to us first asking about them.
It seems like we'd be better off adjusting our test suite to work around the constraints or disable them only for the duration of a test intentionally violating them?
[-]test_httplib test_nntplib test_ssl fail on ARMv7 Ubuntu 3.7 and ARMv7 Ubuntu 3.x buildbots[/-][+]test_httplib test_nntplib test_ssl fail on ARMv7 buster/sid buildbots[/+][-]test_httplib test_nntplib test_ssl fail on ARMv7 buster/sid buildbots[/-][+]test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster buildbot[/+]vstinner commentedon Feb 15, 2019
Does test_ssl pass on the master branch?
gpshead commentedon Feb 15, 2019
Not on this debian buster bot. look back a couple comments, there is a workaround. it seems to be an OpenSSL configuration issue / test expectations issue.
I think we should ultimately get our test suite so that it passes in default OS distro OpenSSL configs.
That could mean any of an altered environment for some tests, or skipping some tests in such an environment, or changing some tests to fit within modern OpenSSL desired ciphersuite/protocol setting constrains constraints - or a mix of all three.
[-]test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster buildbot[/-][+]test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a)[/+]gpshead commentedon Feb 15, 2019
release managers are free to defer this blocker. i'm just marking it as such for the purposes of making sure it is a conscious decision.
The problem is more likely with our test suite vs the environment than it is with CPython itself.
gpshead commentedon Feb 15, 2019
FWIW I've just manually confirmed that running Python 2.7's test_ssl with OPENSSL_CONF=/invalid-path set passes on the debian buster buildbot host.
18 remaining items
JulienPalard commentedon May 8, 2019
I'm still seeing the issue on #12255 (freshly rebased to master to have 2cc0223.
On this build: https://dev.azure.com/Python/cpython/_build/results?buildId=42065
======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------
which does not looks covered by 2cc0223 which only checks for key too weak.
gpshead commentedon May 8, 2019
thats https://bugs.python.org/issue36816 (separate issue as our infrastructure is fixed to have a modern certificate). PR pending automerge post-CI.
JulienPalard commentedon May 8, 2019
👍
gpshead commentedon May 11, 2019
In our 3.6 tree the test_ssl failure is now:
======================================================================
ERROR: test_protocol_sslv23 (test.test_ssl.ThreadedTests)
Connecting to an SSLv23 server with various client options
----------------------------------------------------------------------
======================================================================
ERROR: test_protocol_tlsv1_1 (test.test_ssl.ThreadedTests)
Connecting to a TLSv1.1 server with various client options.
----------------------------------------------------------------------
gpshead commentedon May 11, 2019
(same on 2.7)
gpshead commentedon May 13, 2019
New changeset 7346a16 by Gregory P. Smith in branch '2.7':
[2.7] bpo-35925: Skip SSL tests that fail due to weak external certs or old TLS (GH-13124) (GH-13253)
7346a16
gpshead commentedon May 13, 2019
3.6 (and 3.5 if larry wants) are the only remaining trees to apply this to, assigning to the 3.6 RM.
ned-deily commentedon May 29, 2019
New changeset 8ab624b by Ned Deily (Gregory P. Smith) in branch '3.6':
[3.6] bpo-35925: Skip SSL tests that fail due to weak external certs or old TLS (GH-13124) (GH-13252)
8ab624b