Repository navigation
[CVE-2024-11168] urlparse incorrectly retrieves IPv4 and regular name hosts from inside of brackets #103848
Description
Activity
- added a commit that references this issue
on May 10, 2023 - added a commit that references this issue
on May 10, 2023 As implemented, the URL
scheme://foobar.[::1]/is considered correct, but results in thehostnamevalue of::1. The hostname portion of the netloc is invalid however, as it doesn't start with[. Shouldn't the parser at least validate that the[bracket is the first character or directly following a@?This change caused an issue on our system. We have brackets in a password. Since 3.11.4, urlparse tries to parse an ip address if it finds brackets. The brackets don't even have to be in the correct order.
Running this code:
from urllib.parse import urlparse urlparse("https://user:some]password[@host.com")
In 3.11.3 it works as expected:
ParseResult(scheme='https', netloc='user:some]password[@host.com', path='', params='', query='', fragment='')In 3.11.4 it throws an exception:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.11/urllib/parse.py", line 395, in urlparse splitresult = urlsplit(url, scheme, allow_fragments) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/parse.py", line 500, in urlsplit _check_bracketed_host(bracketed_host) File "/usr/local/lib/python3.11/urllib/parse.py", line 446, in _check_bracketed_host ip = ipaddress.ip_address(hostname) # Throws Value Error if not IPv6 or IPv4 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/ipaddress.py", line 54, in ip_address raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address') ValueError: '@host.com' does not appear to be an IPv4 or IPv6 address
This change caused an issue on our system. We have brackets in a password. Since 3.11.4, urlparse tries to parse an ip address if it finds brackets. The brackets don't even have to be in the correct order.
Running this code:
from urllib.parse import urlparse urlparse("https://user:some]password[@host.com")
In 3.11.3 it works as expected:
ParseResult(scheme='https', netloc='user:some]password[@host.com', path='', params='', query='', fragment='')That this worked before was a bug.
urlparse()was violating the various and miriad URL standards before;[and]are reserved characters with meaning in the netloc portion of a URL (the part between thescheme://and/path) and must be URL encoded. See the WhatWG living URL standard percent-encoding section; for the userinfo part (username + password), you need to encode:- U+002F (
/), U+003A (:), U+003B (;), U+003D (=), U+0040 (@), U+005B ([) to U+005E (^), inclusive, and U+007C (|). - U+003F (
?), U+0060 (`), U+007B ({), and U+007D (}) (path percent-encode set). - U+0020 SPACE, U+0022 (
"), U+0023 (#), U+003C (<), and U+003E (>) (query percent-encode set). - U+0000 NULL to U+001F INFORMATION SEPARATOR ONE, inclusive, and all codepoints past U+007E (
~) (C0 control percent-encode set)
So this works:
>>> from urllib.parse import urlparse >>> urlparse("https://user:some%5Dpassword%5B@host.com") ParseResult(scheme='https', netloc='user:some%5Dpassword%5B@host.com', path='', params='', query='', fragment='') >>> _.password 'some%5Dpassword%5B'The password is still URL encoded, use
urllib.parse.unquote()for that.Further references:
- Wikipedia: URL encoding
- RFC 3986, section 2.2 Reserved Characters
Reacted by Marco Glauser, Gregory P. Smith and RacerZ- U+002F (
Thank you for the clarification! This makes a lot of sense :)
- added a commit that references this issue
on Jun 15, 2023 I'm also hitting "ValueError: 'xxxxx' does not appear to be an IPv4 or IPv6 address" on code that was working.
In my case I have no control over the input data, it's background noise from the internet that i'm parsing and analyzing. Some of the crap in my dataset is very much not standards compliant. None the less, as it was working and now isn't i figured i should 👍 before i start vendoring old versions of the stdlib into my project 😆
- added a commit that references this issue
on Aug 9, 2023 21 remaining items
- added 5 commits that reference this issue
on Jan 21, 2025 - added a commit that references this issue
on Jan 22, 2025 My backport of fix for Python 3.6 (which with minimal changes applies to 3.4 as well, and with a lot of changes, including injecting
ipaddressfrom https://github.com/phihag/ipaddress into the Python standard library, into 2.7) is at openSUSE-Python@4f2496bHi @mcepl. I'm currently evaluating how to fix this in older Debian dists for 2.7, and FYI it seems the version from OpenSUSE Leap 15 (150000.74.1) doesn't fully fix it:
$ PYTHONPATH=/root/rpmbuild/BUILD/Python-2.7.18/Lib \ python2 -c 'import urlparse; print(urlparse.urlparse("https://user:some]password[@host.com"))' ParseResult(scheme='https', netloc='user:some]password[@host.com', path='', params='', query='', fragment='')(expected: ValueError)
It seemsCVE-2024-11168-validation-IPv6-addrs.patchfixes the "http://" special case but not the general case (which handles "httpS://"). I didn't check the 3.6 patch closely but it seems it's the opposite situation there.After fixing the above I also get a couple test_urlparse errors, related to passing non-unicode data to the ipaddress backport. Casting to Unicode
ipaddress.ip_address(unicode(hostname))appears to tame the testsuite.(expected: ValueError) It seems
CVE-2024-11168-validation-IPv6-addrs.patchfixes the "http://" special case but not the general case (which handles "httpS://"). I didn't check the 3.6 patch closely but it seems it's the opposite situation there.After fixing the above I also get a couple test_urlparse errors, related to passing non-unicode data to the ipaddress backport. Casting to Unicode
ipaddress.ip_address(unicode(hostname))appears to tame the testsuite.Do you have a corrected patch? I am all ears!
Do you have a corrected patch? I am all ears!
Now yes :)
OpenSUSE patch with corrections: https://salsa.debian.org/lts-team/packages/python2.7/-/blob/08639c6a31c4726e2862d2908a8a805be9c186df/debian/patches/CVE-2024-11168-opensuse.patch
Alternate approach, without embedding (and having to maintain) ipaddress: https://salsa.debian.org/lts-team/packages/python2.7/-/blob/c62e4e05f86c0e14ae6aa405382268d3c97137c6/debian/patches/CVE-2024-11168.patch- On Sat Mar 8, 2025 at 9:20 AM CET, Beuc wrote: Now yes :) OpenSUSE patch with corrections: https://salsa.debian.org/lts-team/packages/python2.7/-/blob/08639c6a31c4726e2862d2908a8a805be9c186df/debian/patches/CVE-2024-11168-opensuse.patch Alternate approach, without embedding (and having to maintain) ipaddress: https://salsa.debian.org/lts-team/packages/python2.7/-/blob/c62e4e05f86c0e14ae6aa405382268d3c97137c6/debian/patches/CVE-2024-11168.patchThank you. In the end I went just with the modification of the original patch, I like how similar the patch is on all versions of the interpreter.
- added a commit that references this issue
on Apr 23, 2025 - added a commit that references this issue
on Jul 4, 2025 - added a commit that references this issue
on Aug 12, 2025 - added a commit that references this issue
on Feb 3, 2026
Background
RFC 3986 defines a host as follows
Where
WhatWG says that "A valid host string must be a valid domain string, a valid IPv4-address string, or: U+005B ([), followed by a valid IPv6-address string, followed by U+005D (])."
The Bug
This is code from
Lib/urllib/parse.py:196-208used for retrieving the hostname from the netlocIt will incorrectly retrieve IPv4 addresses and regular name hosts from inside brackets. This is in violation of both specifications.
Minimally reproducible example:
Your environment
23cf1e2)Linked PRs