Repository navigation
Making urlparse WHATWG conformant #88049
Description
| BPO | 43883 |
|---|---|
| Nosy | @gpshead, @orsenthil, @vstinner, @serhiy-storchaka, @mlissner, @tirkarthi |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/orsenthil'
closed_at = None
created_at = <Date 2021-04-18.19:43:37.075>
labels = ['type-bug']
title = 'Making urlparse WHATWG conformant'
updated_at = <Date 2021-05-05.01:34:26.506>
user = 'https://github.com/orsenthil'bugs.python.org fields:
activity = <Date 2021-05-05.01:34:26.506>
actor = 'gregory.p.smith'
assignee = 'orsenthil'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-04-18.19:43:37.075>
creator = 'orsenthil'
dependencies = []
files = []
hgrepos = []
issue_num = 43883
keywords = []
message_count = 4.0
messages = ['391344', '391347', '391427', '392969']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'orsenthil', 'vstinner', 'serhiy.storchaka', 'Mike.Lissner', 'xtreak']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43883'
versions = []Metadata
Metadata
Assignees
Projects
- Status
Activity
orsenthil commentedon Apr 18, 2021
Mike Lissner reported that a set test suites that exercise extreme conditions with URLs, but in conformance with url.spec.whatwg.org
was maintained here:
https://github.com/web-platform-tests/wpt/tree/77da471a234e03e65a22ee6df8ceff7aaba391f8/url
These test cases were used against urlparse and urljoin method.
https://gist.github.com/mlissner/4d2110d7083d74cff3893e261a801515
Quoting verbatim
serhiy-storchaka commentedon Apr 18, 2021
It would be interesting to test also with the yarl module. It is based on urlparse and urljoin, but does extra normalization of %-encoding.
vstinner commentedon Apr 20, 2021
See also bpo-43882.
gpshead commentedon May 5, 2021
FWIW rather than implementing our own URL parsing at all... wrapping a library extracted from a compatible-license major browser (Chromium or Firefox) and keeping it updated would avoid disparities.
Unfortunately, I'm not sure how feasible this really is. Do all of the API surfaces we must support in the stdlib for compatibility's sake with urllib line up with such a browser core URL parsing library?
Something to ponder. Unlikely something we'll actually do.
xmo-odoo commentedon Jun 15, 2023
If
urllib.parseis to be tested for whatwg conformance, shouldn't it beurlsplit? And maybeurlparseshould finally be deprecated?urlparse's behaviour was superseded by RFC 2396 back in 1998 (which is what resulted in the addition of urlsplit: #35466), the whatwg url standard is a descendant of that (itself intending to supersede RFC 3986). As of the url standard (and already RFC 3986) the entire concept of path segment parameter has been dropped.
By its very nature and purpose, urlparse can't pass the conformance suite, because any time it encounters a
;within a path segment it's going to move data out of the pathname which it should not. And the suite likely has multiple test cases where the path contains;characters.idahogray commentedon Nov 14, 2024
I'm not sure if this is the correct issue or not, but it seemed related. Chromium (and derivative browsers) recently implemented a change to URL parsing that broke a bunch of custom protocols. We have an application that uses the protocol
pw://which is broken by this change. We have a django application that was using the django URLValidator to validate those custom URLs.The way I understand it, the correct format for custom protocols is to not have the
//. I don't believe urlsplit supports these because it is looking for the//in order to extract the netloc. At least it is in 3.12, I haven't looked at 3.13.Thanks and please let me know if this is the wrong issue or if you need more information.
gpshead commentedon Nov 14, 2024
also @sethmlarson as FYI - it isn't clear we'd ever even be able to do what this issue suggests or that we should FWIW. there are multiple competing interests in the world who have valid needs to use URLs (URIs?) in different ways.
sethmlarson commentedon Nov 14, 2024
I don't think we should chase browsers, and therefore we shouldn't implement WHATWG. They are allowed to update their "fleet" globally on a whim, they primarily deal with users physically typing things into address bars, we have things called backwards compatibility guarantees. I think implementing RFC 3986 is appropriate for software like Python.