Repository navigation
SSL lib does not handle trailing dot (period) in hostname or certificate #76178
Description
Activity
samiam commented
on Nov 10, 2017 samiammannequinMannequinAuthorI recently came across an issue in the ssl library and have a simple fix to address it.
When doing hostname verification against an X.509 certificate, a trailing dot (period) in the hostname is matched against the certificate. But the trailing dot should only be applied to the DNS lookup not the certificate match.
Conversely, a certificate that has a trailing dot in its commonName (probably rare but allowed) should match a hostname without the trailing dot. As the ssl library is written now, both cases fail.
The truth table below shows the current python ssl DNS matching behavior.
+----------------------------------------+
| # hostname certificate MATCH |
| +------------------------------------+ |
| | dns dns. cname cname. | |
| +------------------------------------+ |
| 1 x x TRUE |
| 2 x x FALSE |
| 3 x x FALSE |
| 4 x x TRUE |
+----------------------------------------+Case 1 and 4 currently match as both hostname and certificate strings match exactly when the trailing dot is either present or not.
Case 2 is unlikely, as certificates are rarely signed with a trailing dot in the subject commonName and if they were, clients would ALWAYS have to enter the hostname with a trailing dot to get a match.
Case 3 is more likely where the hostname has a trailing dot, but the certificate does not. For example, "www.example.com." is used for the DNS lookup, but then, "www.example.com." will not match the certificate due to the trailing dot missing from the certificate.
I propose the truth table should be true in all cases and just ignore the trailing dot in both the hostname and certificate.
As best I can tell, the RFCs [1] are silent on this issue. Although the hostname and commonName strings currently must match, there are a couple of precedents where ignoring the trailing dot is done in practice.
Web browsers allow a trailing dot in the URI and will accept a certificate even when the certificate doesn't have a trailing dot. For example, visit Google with trailing dot (https://www.google.com./) from a browser of your choice and check certificate. It should check out as valid. [2]
Also, at least two language SSL libraries, Ruby [3] and Go [4], match certificates when hostnames contain a trailing dot. Lastly cURL [5] ignores trailing dots in certs and hostnames.
In summary, I don't feel the current python ssl library is wrong - it is following an interpretation of the RFC. But I think it can be more permissive, following the spirit of the RFC without sacrificing security.
Patch attached with code change. If accepted, I can do a more formal PR, backport to 2.7 and add tests.
Thoughts?
-SamExample of issue
================Python 3.7.0a2+ (heads/master-dirty:cbe1756e3e, Nov 3 2017, 15:56:14) [Clang 8.1.0 (clang-802.0.42)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import ssl >>> cert = {'subject': ((('commonName', 'example.com'),),)} >>> ssl.match_hostname(cert, "example.com") ## Case 1 from truth table >>> ssl.match_hostname(cert, "example.com.") ## Case 3 from truth table Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".../cpython/Lib/ssl.py", line 330, in match_hostname % (hostname, dnsnames[0])) ssl.CertificateError: hostname 'example.com.' doesn't match 'example.com' >>> cert = {'subject': ((('commonName', 'example.com.'),),)} >>> ssl.match_hostname(cert, "example.com.") ## Case 4 from truth table >>> ssl.match_hostname(cert, "example.com") ## Case 2 from truth table Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".../cpython/Lib/ssl.py", line 330, in match_hostname % (hostname, dnsnames[0])) ssl.CertificateError: hostname 'example.com' doesn't match 'example.com.'
References
==========[1] RFCs - there may be other RFCs addressing X.509 certificates
https://tools.ietf.org/html/rfc6125
https://tools.ietf.org/html/rfc5280
https://tools.ietf.org/html/rfc3986[2] Old Mozilla thread discussing the trailing dot:
https://bugzilla.mozilla.org/show_bug.cgi?id=134402#c36
I quote:
Yes, it's ok to match "www.example.com." (trailing) to the cert with "www.example.com" (no trailing)
It's also OK to match "www.example.com" (no trailing) to the cert with "www.example.com." (trailing)[3] Ruby
Ironically Ruby doesn't even take the trailing dot into consideration as it splits the strings using dot as the delimiter.
irb> "www.example.com".split('.') == "www.example.com.".split('.')
=> truehttps://github.com/ruby/openssl/blob/master/lib/openssl/ssl.rb#L295
[4] Go
The trailing dot is explicitly removed in Go TLS library.
https://github.com/golang/go/blob/master/src/crypto/x509/verify.go#L913
[5] cURL
cURL strips trailing dot following behavior in browsers.
https://github.com/curl/curl/blob/master/lib/hostcheck.c#L51
The cURL author discusses the trailing dot in a couple threads.
curl/curl#716
https://lists.w3.org/Archives/Public/ietf-http-wg/2016JanMar/0430.htmlIn the future Python will no longer use its own hostname verification code. Instead we are going to rely on OpenSSL to verify the hostname for us. A trailing dot also affects SNI. How do OpenSSL's functions SSL_set_tlsext_host_name() and X509_VERIFY_PARAM_set1_host() deal with a trailing dot?
How do TLS servers such as Apache mod_ssl, Apache mod_nss, nginx, Go's TLS server, ... deal with trailing dot in SNI?
Trailing dots in hostname seem to be protocol specific, e.g. SMTP does not allow them. Unless you find a RFC that mandates support for trailing dots in TLS, I'm against a change in Python's TLS stack. It's too risky to mess up SNI, too.
I'd rather follow RFC 5890, make the caller deal with FQDN + trailing dot and require libraries to pass in a DNS Domain Names (a fully qualified domain name without a trailing dot) to server_hostname.
https://tools.ietf.org/html/rfc6125#section-2.2
https://tools.ietf.org/html/rfc5890#section-2.2(The complete name convention using a trailing dot described
in RFC 1123 [RFC1123], which can be explicit as in "www.example.com."
or implicit as in "www.example.com", is not considered in this
specification.)After more investigation and discussion with Daniel Stenberg, I'm considering to close the issue as WONTFIX + documentation update. The issue cannot be addressed in the SSL/TLS layer. I'm waiting for a reply from Ryan Sleevi on CAB baseline requirements. BR 1.5.1 does not state if trailing dots are allowed.
The trailing dot issue affects more than just hostname matching. For HTTPS, server name indication (SNI) and HTTP "Host" header play an important role, too. In general the SNI TLS header and HTTP Host header must match. In case the HTTP header is missing or deviates from the SNI header, web servers like Apache fail with Bad Request error. In general SNI must also match a SAN dNSName extension.
Apache with mod_ssl strips off trailing dots internally. Daniel pointed out that other webservers (IIS) do not handle trailing dots correctly. Some protocols like SMTP do not allow trailing dot in FQDN.
IMO the problem should be handled in high level libraries such as urllib. urllib should use the FQDN with trailing dot for DNS resolution, then strip off the trailing dot and use the FQDN for HTTP Host header and server_hostname.
Ryan said:
Chrome will match both trailing dot and non-trailing dot in URL against non-trailing dot in cert. trailing dot in cert is 5280 violation by not being in preferred name syntaxsamiam commented
on Nov 14, 2017 samiammannequinMannequinAuthorSorry I wasn't able to get back to you sooner.
If having a trailing dot in the cert is an RFC violation, then case 2 can be left alone.
As for case 3, we can be more explicit: if hostname ends in a dot AND cert does not end in a dot, strip dot from hostname. This seems to be what Ryan was saying Chrome does.
I did a test using s_client in openssl. Testing all 4 cases in the truth table returned 200s.
$ openssl s_client -connect www.google.com.:443 ... # Enter next two lines and press return twice HEAD / HTTP/1.0 Host: www.google.com.# Returns 200
HTTP/1.0 200 OK
Date: Sat, 11 Nov 2017 21:20:44 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
...So it would appear openssl against Google handles dots ok, but I could be wrong. I don't know what server software they are running.
As for testing other server ssl implementations what are you looking for?
I found a small C openssl client implementation.
https://ubuntuforums.org/showthread.php?t=2217101&p=12989750#post12989750
Compiling that code with some minor tweaks against openssl and testing it with different hostnames and Host headers (dot and no dot), the ssl connection was established and data read. Invalid constructs led to errors.
Yes, you could move the logic to urllib, but I'm not sure it's practical as many folks just expect the ssl lib to handle the nuances. If users have to handle it themselves or include urllib, it seems like an extra lift.
I appreciate you taking the time to consider the issue.
As I explained before, the ssl module is the wrong place to address the issue. You *must* keep SNI TLS extension, HTTP Host header, and hostname for SAN matching in sync. Python uses the server_hostname argument for both SNI and hostname verification.
The issue must be solved in HTTP layer because the HTTP layer is the only place that can affect the HTTP Host header and SNI.
OpenSSL and NSS (Firefox's crypto and TLS lib) agree with me. Both don't like trailing dots in hostname either. BoringSSL's hostname verification code is based on OpenSSL's code. I'm pretty sure that Chrome handles trailing dot in a different layer, not in the actual TLS and X.509 handler. Ryan merely said that Chrome supports hostnames with trailing dot, not BoringSSL.
$ /usr/lib64/nss/unsupported-tools/vfyserv www.python.org Connecting to host www.python.org (addr 151.101.112.223) on port 443 Handshake Complete: SERVER CONFIGURED CORRECTLY bulk cipher AES-128-GCM, 128 secret key bits, 128 key bits, status: 1 subject DN: CN=www.python.org,O=Python Software Foundation,L=Wolfeboro,ST=New Hampshire,C=US,postalCode=03894-4801,STREET=16 Allen Rd,serialNumber=3359300,incorporationState=Delaware,incorporationCountry=US,businessCategory=Private Organization issuer DN: CN=DigiCert SHA2 Extended Validation Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US 0 cache hits; 0 cache misses, 0 cache not reusable ***** Connection 1 read 518 bytes total. $ /usr/lib64/nss/unsupported-tools/vfyserv www.python.org. Connecting to host www.python.org. (addr 151.101.112.223) on port 443 Error in function PR_Write: -12276 - Unable to communicate securely with peer: requested domain name does not match the server's certificate. $ openssl s_client -servername www.python.org -verify_hostname www.python.org -connect www.python.org:443 ... SSL handshake has read 4204 bytes and written 403 bytes Verification: OK Verified peername: www.python.org ... $ openssl s_client -servername www.python.org. -verify_hostname www.python.org. -connect www.python.org.:443 ... SSL handshake has read 4204 bytes and written 404 bytes Verification error: Hostname mismatch ...
I'm closing this bug as "not a bug" because it works as intended. The trailing dot has to be handled in the application layer.
I'd like to petition to have this issue re-opened as a bug.
Here are my reasons:
- The RFC on TLS extension that defines SNI explicitly states that trailing dots should not be sent in SNI ("The hostname is represented as a byte string using ASCII encoding without a trailing dot.")
- OpenSSL is out of spec according to the RFC when it sends a HostName with a trailing dot as SNI
- In OpenSSL's open issue on this matter, the assertion that the application layer should handle this is made, but in the case of python, _ssl.c is the application layer (it sets SNI for TLS requests based on the request being made if the request has a domain name but no SNI is explicitly set), so when internal python libraries call into OpenSSL the current expectation from OpenSSL is that calls to their functions should strip trailing dots.
- When making a request to something like "example.com." where you shouldn't have to manually set SNI, in order for this to work the client code must manually set SNI to "example.com" because neither python nor OpenSSL are following spec.
- FQDNs are beneficial for explicit DNS resolution but problematic for TLS the way thing are right now. While there are work arounds the current behavior of python (and OpenSSL) is unexpected and requires investment from developers to manually override the SNI that python would have sent with the malformed HostName including a trailing dot.
Refs:
openssl/openssl#11560
https://datatracker.ietf.org/doc/html/rfc6066#section-3The way I see the options here, the choices are:
- remain out of spec according to RFC 6066
- never automatically set SNI and always require SNI be set by the caller (breaks back compat)
- strip trailing dots when automatically generating SNI for ssl hostname verification
If you have any concerns that aren't addressed I'd love to continue this discussion. I'm also happy to submit a patch.
Reacted by Gustavo Arzola and Matt McShane
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: