Repository navigation
Use getaddrinfo() in urllib2.py for IPv6 support #44672
Description
Activity
dcantrell-rh commented
on Mar 7, 2007 dcantrell-rhmannequinMannequinAuthorA number of base Python modules use gethostbyname() when they should be using getaddrinfo(). The big limitation hit when using gethostbyname() is the lack of IPv6 support.
This first patch is for urllib2.py. It replaces all uses of gethostbyname() with getaddrinfo() instead. getaddrinfo() returns a 5-tuple, so additional code needs to wrap a getaddrinfo() call when replacing gethostbyname() calls. Still should be pretty simple to read.
I'd like to see this patch added to the next stable release of Python, if at all possible. I am working up patches for the other modules I see in the Lib/ subdirectory that could use getaddrinfo() instead of gethostbyname().
jjlee commented
on Jul 11, 2007 jjleemannequinMannequin- Where are the tests? A functional test, perhaps in test_urllib2net.py, for IPv6 support in urllib2 would be especially welcome, I think.
- Why does .check_host() not begin with an underscore?
- "check_host" is a poor name. How about "_is_localhost"?
- locals is a built-in function, hence usually considered good style not to use it as a name.
- Is is necessary to call make_host_tuple(searchlist) twice?
- The patch appears to fix several bugs at once (e.g. adding a try: / except: suite around a large part of an existing method to catch socket.error).
Hi,
The patch attached required a complete rewrite. I am attaching the
modified patch, which will just substitute socket.gethostbyname with a
function gethost_addrinfo which internally uses getaddrinfo and takes
care of the IPv4 or IPv6 addresses translation.jjlee, skip: let me know your comments on this.
One note we have to keep in mind is, testing on IPv6 address.
For eg. on my system /etc/hosts
10.98.1.6 goofy.goofy.com
#fe80::219:5bff:fefd:6270 localhost
127.0.0.1 localhosttest_urllib2 will PASS for the above.
But if I uncomment the IPv6 address, opening the local file fails. I am
not sure how local file access is done with IPv6 and should urllib2
(local file opening function) itself needs to be modified. Shall check
into that, with next version.What I don't understand here is... if gethostbyname() lacks of IPv6
support, instead of creating a new function why not to add the
functionality to that same function?Right now gethostbyname() is implemented in C, which would be the
drawback of making it a Python function?dmorr commented
on Jan 1, 2009 dmorrmannequinMannequinSenthil,
I don't think your gethost_addrinfo() function will work. On a v6-
enabled machine, it will only return v6 or v4 names. Shouldn't it
return both (since a machine could have both v4 and v6 addresses)? For
example, on my machine, I have the following addresses for
"localhost": ::1, fe80::1%lo0, 127.0.0.1.Also, why is the AI_CANONNAME flag set? The canonname field isn't used.
And you only appear to take the last IP address returned (sa[0]).
Shouldn't you return all the addresses?dmorr commented
on Jan 1, 2009 dmorrmannequinMannequinQuestion: Why does FTPHandler.ftp_open() try to resolve the hostname()?
The hostname will be passed into connect_ftp(), then into
urllib.ftpwrapper(), and eventually into ftplib.FTP.connect(), which is
IPv6-aware.Derek,
This patch was along the lines that when IPv6 address is present, return
the first address,which I assumed to be active address and would make
the urllib2 work.I am not sure, if returning all the addresses would help and how would
we define which address to use?AI_CANONNAME flag, I don't accurately remember it now. But I had
encountered issues when testing on IPv-4 systems without it.I am having different opinion on this issue now.
First is, taking from Facundo's comment on having this functionality in
gethostbyname() and implementing it in C.Second is, the wrapper function and suitable way needs to be defined.
I am sorry, I fail to understand the question on why ftp_open does
hostname resolution. You mean to say without it, if we pass it to
ftplib.FTP.connect() it would work for IPv6 address?dmorr commented
on Jan 2, 2009 dmorrmannequinMannequinMy understanding is that the FileHandler checks if the file:// URL
contains the hostname or localhost IP of the local machine (isn't that
what FileHandler.names is for?). So, shouldn't the following URLs all
open the same file:file:///foo.txt
file://localhost/foo.txt
file://127.0.0.1/foo.txt
file://[::1]/foo.txtIf that is the case, then doesn't FileHandler.names need to have all of
those values in it?I am a little confused by this though. It looks like
FileHandler.file_open() checks if there is a hostname in the URL, and
if so, uses FTPHandler instead. So why does FileHandler.open_local_file
check the hostname value?For your other points, gethostbyname() in libc can only handle IPv4
addresses. The IETF defined the getaddrinfo() interface as an IP
version neutral replacement. I would recommend using getaddrinfo().Yes, FTPHandler creates an urllib.FTPWrapper object. That object calls
into ftplib, which is already IPv6-capable. So, I don't think we need
to do hostname resolution in FTPHandler.I am a little confused by this though. It looks like
FileHandler.file_open() checks if there is a hostname in the URL, and
if so, uses FTPHandler instead. So why does FileHandler.open_local_file
check the hostname value?You are right. Even I had observed this, but did not dispute it. Let
me try to look into the history to see why it so. Perhaps it needs to
change.For your other points, gethostbyname() in libc can only handle IPv4
addresses. The IETF defined the getaddrinfo() interface as an IP
version neutral replacement. I would recommend using getaddrinfo().
Yes, FTPHandler creates an urllib.FTPWrapper object. That object calls
into ftplib, which is already IPv6-capable. So, I don't think we need
to do hostname resolution in FTPHandler.Thanks for the info. I shall look into both in revision of the path.
- using getaddrinfo() for IP version neutral call.
- passing the hostname directly to ftplib. ( I am not sure of
consequences, need to investigate).
BreamoreBoy commented
on Sep 16, 2010 BreamoreBoymannequinMannequin@senthil should this be assigned to your good self?
RamchandraApte commented
on Jul 20, 2012 RamchandraAptemannequinMannequinBump.
The original complaint is long fixed: the HTTP(S) path stopped depending on
gethostbynamewhenhttp.clientmoved tosocket.create_connection(getaddrinfo, IPv6-capable) —do_openjust passes the hostname through. Of thegethostbynamecalls still inLib/urllib/request.py,localhost()/thishost()are unused legacy helpers (only a test callsthishost()), and theFileHandler/_is_local_authorityuses are local self-checks. The one real residual isFTPHandler.ftp_open: it pre-resolves the host withsocket.gethostbyname(IPv4-only) before handing it toftplib.FTP.connect, which resolves viacreate_connectionanyway — so an IPv6-only FTP host fails at that line while plainftplibworks. Derek flagged this exact spot in 2009. The fix is dropping the pre-resolution, not converting it; worth keeping open for that. The patch is up as #154192.Why gethostbyname is the IPv4 bottleneck (current main, 6df4993)
import socket, sys print(sys.version) try: socket.gethostbyname("::1") except socket.gaierror as e: print("gethostbyname('::1') ->", type(e).__name__, e) print("getaddrinfo('::1', 21)[0][:2] ->", socket.getaddrinfo("::1", 21, type=socket.SOCK_STREAM)[0][:2])
3.16.0a0 (remotes/upstream/HEAD:6df4993bb64, Jul 19 2026, 20:24:53) [Clang 21.0.0 (clang-2100.1.1.101)] gethostbyname('::1') -> gaierror [Errno 8] nodename nor servname provided, or not known getaddrinfo('::1', 21)[0][:2] -> (<AddressFamily.AF_INET6: 30>, <SocketKind.SOCK_STREAM: 1>)
Metadata
Metadata
Assignees
Projects
- Status
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs