ssl.match_hostname(): sub string wildcard should not match IDNA prefix

@loewis

BPO	17997
Nosy	@loewis, @warsaw, @birkenfeld, @pitrou, @larryhastings, @tiran, @jwilk, @abadger, @bdarnell
Files	match_hostname_RFC6125.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-09-30.12:49:13.940>
created_at = <Date 2013-05-17.14:04:53.841>
labels = ['type-security', 'release-blocker']
title = 'ssl.match_hostname(): sub string wildcard should not match IDNA prefix'
updated_at = <Date 2014-09-30.12:49:13.937>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2014-09-30.12:49:13.937>
actor = 'georg.brandl'
assignee = 'none'
closed = True
closed_date = <Date 2014-09-30.12:49:13.940>
closer = 'georg.brandl'
components = []
creation = <Date 2013-05-17.14:04:53.841>
creator = 'christian.heimes'
dependencies = []
files = ['31245']
hgrepos = []
issue_num = 17997
keywords = ['patch']
message_count = 16.0
messages = ['189454', '189514', '189516', '189527', '189909', '189928', '194950', '195058', '196823', '200724', '201422', '201431', '203159', '207172', '222211', '227895']
nosy_count = 13.0
nosy_names = ['loewis', 'barry', 'georg.brandl', 'pitrou', 'larry', 'christian.heimes', 'jwilk', 'a.badger', 'Arfrever', 'BreamoreBoy', 'python-dev', 'Ben.Darnell', 't-8ch']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue17997'
versions = ['Python 3.3', 'Python 3.4']

Python's ssl.match_hostname() does sub string matching as specified in RFC 2818:

Names may contain the wildcard
character * which is considered to match any single domain name
component or component fragment. E.g., *.a.com matches foo.a.com but
not bar.foo.a.com. f*.com matches foo.com but not bar.com.

The RFC doesn't specify how internationalized domain names shoould be handled because it predates RFC 5890 for IDNA by many year. IDNA are prefixed with "xn--", e.g. u"götter.example.de".encode("idna") ==
"xn--gtter-jua.example.de". This can result into false positive matches for a rule like "x*.example.de".

Chrome has special handling for IDN prefix in X509Certificate::VerifyHostname()
http://src.chromium.org/viewvc/chrome/trunk/src/net/cert/x509_certificate.cc

Also see bpo-17980

Actually, I don't this is a bug: match_hostname() expects str data, and therefore IDNA-decoded domain names:

>>> b"xn--gtter-jua.example.de".decode("idna")
'götter.example.de'

Doing the matching on the decoded domain name should be safe.
Then it very much depends on whether the data you've got was IDNA-decoded, or naïvely ASCII-decoded, and I don't think the Python stdlib is very consistent here. Looking at the socket module, gethostbyaddr and getnameinfo seem to use ASCII decoding...

It's called "internationalized domain name for APPLICATIONS". ;) It's up to the application to interpret the ASCII text as IDNA encoded FQDNs. As far as I know DNS, SSL's CNAME and OS interfaces etc. always use ASCII labels. It's an elegant solution. Just the UI part of an application needs to understand IDNA.

http://tools.ietf.org/html/rfc6125#section-6.4.2

If the DNS domain name portion of a reference identifier is an
internationalized domain name, then an implementation MUST convert
any U-labels [IDNA-DEFS] in the domain name to A-labels before
checking the domain name. In accordance with [IDNA-PROTO], A-labels
MUST be compared as case-insensitive ASCII. Each label MUST match in
order for the domain names to be considered to match, except as
supplemented by the rule about checking of wildcard labels
(Section 6.4.3; but see also Section 7.2 regarding wildcards in
internationalized domain names).

Coincidentally the same RFC contains matching rules for wild card certs
http://tools.ietf.org/html/rfc6125#section-6.4.3

If a client matches the reference identifier against a presented
identifier whose DNS domain name portion contains the wildcard
character '*', the following rules apply:

The client SHOULD NOT attempt to match a presented identifier in
which the wildcard character comprises a label other than the
left-most label (e.g., do not match bar.*.example.net).
If the wildcard character is the only character of the left-most
label in the presented identifier, the client SHOULD NOT compare
against anything but the left-most label of the reference
identifier (e.g., *.example.com would match foo.example.com but
not bar.foo.example.com or example.com).
The client MAY match a presented identifier in which the wildcard
character is not the only character of the label (e.g.,
baz*.example.net and *baz.example.net and b*z.example.net would
be taken to match baz1.example.net and foobaz.example.net and
buzz.example.net, respectively). However, the client SHOULD NOT
attempt to match a presented identifier where the wildcard
character is embedded within an A-label or U-label [IDNA-DEFS] of
an internationalized domain name [IDNA-PROTO].

It's called "internationalized domain name for APPLICATIONS". ;) It's
up to the application to interpret the ASCII text as IDNA encoded
FQDNs. As far as I know DNS, SSL's CNAME and OS interfaces etc. always
use ASCII labels. It's an elegant solution. Just the UI part of an
application needs to understand IDNA.

The socket module already decodes to/encodes from IDNA in places (e.g. gethostname()). We need a consistent policy in the stdlib; I would like Martin's advice on this.

I finally found the correct RFC for wildcard matching. I think our implementation violates some recommendations.

http://tools.ietf.org/html/rfc6125#section-6.4.2
http://tools.ietf.org/html/rfc6125#section-6.4.3

As a policy, the standard library should accept non-ASCII host names ("U-labels") wherever possible. I.e the hostname parameter of match_hostname should allow for U-labels (as well as A-labels).

When returning names, it should always return the data "as-is", which typically means A-labels. Anybody wanting to display U-labels will need to decode them explicitly.

I believe that the matching of IDNA names doesn't currently happen according to 6.4.2 of RFC 6125, however, this is not actually the issue that Christian reported (which was only about wildcard matching). I suggest to create a separate issue for that.

As for 6.4.3: I find the text to be quite ill-formulated. Specifically, I'm referring to the sentence

   However, the client SHOULD NOT
   attempt to match a presented identifier where the wildcard
   character is embedded within an A-label or U-label [IDNA-DEFS] of
   an internationalized domain name [IDNA-PROTO].

First, in the context of X.509, a wildcard *cannot* be embedded "with an ... U-label"; the certificate can only possibly contain A-labels (because the datatype of dNSName is IA5String).

Second, as written, it *does* allow to match 'götter.example.de' against "x*.example.de", since "x*.example.de" is not an A-label. An A-label is defined as

   An "A-label" is the ASCII-Compatible Encoding (ACE, see
  Section 2.3.2.5) form of an IDNA-valid string.  It must be a
  complete label: IDNA is defined for labels, not for parts of them
  and not for complete domain names.  This means, by definition,
  that every A-label will begin with the IDNA ACE prefix, "xn--"
  (see Section 2.3.2.5), followed by a string that is a valid output
  of the Punycode algorithm [RFC3492] and hence a maximum of 59
  ASCII characters in length.  The prefix and string together must
  conform to all requirements for a label that can be stored in the
  DNS including conformance to the rules for LDH labels
  (Section 2.3.1).  If and only if a string meeting the above
  requirements can be decoded into a U-label is it an A-label.

Since an A-label is required to conform to the LDH label syntax, it cannot possibly contain the asterisk (LDH labels can only contain letters, digits, and the hyphen. Hence, the entire requirement is irrelevant (as literally written). They might mean something else, but I cannot guess what it is that they mean.

I disagree with the classification of this issue as critical. It does not involve a crash, a serious regression, or a breakage of a very important API.

Ryan Sleevi of the Google Chrome Security Team has informed us about another issue that is caused by our failure to implement RFC 6125 wildcard matching rules. RFC 6125 allows only one wildcard in the left-most fragment of a hostname. For security reasons matching rules like *.*.com should be not supported.

For wildcards in internationalized domain names I have followed the piece of advice "In the face of ambiguity, refuse the temptation to guess.". A substring wildcard does no longer match an IDN A-label fragment. '*' still matches a full punycode fragment but 'x*' no longer matches 'xn--foo'. I copied the idea from Chrome's matching code:

http://src.chromium.org/viewvc/chrome/trunk/src/net/cert/x509_certificate.cc?revision=212341#l640

    // * must not match a substring of an IDN A label; just a whole fragment.
    if (reference_host.starts_with("xn--") &&
    !(pattern_begin.empty() && pattern_end.empty()))
    continue;

The relevant RFC section for the patch are

http://tools.ietf.org/html/rfc6125#section-6.4.3
http://tools.ietf.org/html/rfc2818#section-3.1
http://tools.ietf.org/html/rfc2459#section-4.2.1.7
http://tools.ietf.org/html/rfc5280#section-7

Affected versions:

Python 3.2 (< 3.2.5)
Python 3.3 (< 3.3.3)
Python 3.4a1
requests < 1.2.3
https://pypi.python.org/pypi/requests
backports.ssl_match_hostname (<3.2a3)
https://pypi.python.org/pypi/backports.ssl_match_hostname/
urllib3 < 1.6
https://github.com/shazow/urllib3
bzr
setuptools
tornado
pip

So, is this a security issue? I've been wondering if I should apply the attached patch to the backports-ssl_match_hostname module on pypi. I was hoping there'd be some information here as to whether this will be going into the stdlib in the future.

Thus far, ssl_match_hostname has just been a backport of the match_hostname function but if this is a security problem, I could press for us to diverge from the python3 stdlib. It would be easier to make the case if this is seen as a critical problem that will need to be fixed even if the current patch might not be the eventual fix.

Yes, it's a security issue. But the patch would changes the behavior of the function. The current function conforms to RFC 2818. The patch implements RFC 6125, which is more restrictive.

New changeset 10d0edadbcdd by Georg Brandl in branch '3.3':
Issue bpo-17997: Change behavior of ssl.match_hostname() to follow RFC 6125,
http://hg.python.org/cpython/rev/10d0edadbcdd

Also merged to default.

Python 3.2 hasn't been fixed yet.

Should acquire a CVE for the issue?

Just to clarify the status of this issue: it *only* blocks 3.2.

For future reference how do I find out if this has been applied to 3.2?

Since it's been out in 3.2.x for so long, I won't apply this for 3.2 since at this point a behavior change might do more harm than good.

Navigation Menu

Repository navigation

ssl.match_hostname(): sub string wildcard should not match IDNA prefix #62197

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ssl.match_hostname(): sub string wildcard should not match IDNA prefix #62197

Description

Activity

tiran commented on May 17, 2013

pitrou commented on May 18, 2013

tiran commented on May 18, 2013

pitrou commented on May 18, 2013

tiran commented on May 24, 2013

loewis commented on May 24, 2013

tiran commented on Aug 12, 2013

tiran commented on Aug 13, 2013

abadger commented on Sep 3, 2013

tiran commented on Oct 21, 2013

python-dev commented on Oct 27, 2013

birkenfeld commented on Oct 27, 2013

tiran commented on Nov 17, 2013

loewis commented on Jan 2, 2014

BreamoreBoy commented on Jul 3, 2014

birkenfeld commented on Sep 30, 2014

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions