Repository navigation
speed up urllib.request.getproxies_environment #91539
Description
Activity
@bohea The following code reduces the computation time by a factor 3 or more on my system.
def getproxies_environment(): """Return a dictionary of scheme -> proxy server URL mappings. Scan the environment for variables named <scheme>_proxy; this seems to be the standard convention. If you need a different way, you can pass a proxies dictionary to the [Fancy]URLopener constructor. """ # in order to prefer lowercase variables, process environment in # two passes: first matches any, second pass matches lowercase only # select only environment variables which end in (after making lowercase) _proxy candidate_names = [name for name in os.environ.keys() if name[-6:]=='_'] # fast selection of candidates environment = [(name, os.environ[name], name.lower()) for name in candidate_names if name[-6:].lower()=='_proxy'] proxies = {} for name, value, name_lower in environment: if value and name_lower[-6:] == '_proxy': proxies[name_lower[:-6]] = value # CVE-2016-1000110 - If we are running as CGI script, forget HTTP_PROXY # (non-all-lowercase) as it may be set from the web server by a "Proxy:" # header from the client # If "proxy" is lowercase, it will still be used thanks to the next block if 'REQUEST_METHOD' in os.environ: proxies.pop('http', None) for name, value, name_lower in environment: if name[-6:] == '_proxy': if value: proxies[name_lower[:-6]] = value else: proxies.pop(name_lower[:-6], None) return proxiesDoes this work for you? If so, then I will make a PR
- added a commit that references this issue
on Oct 5, 2022 - added 2 commits that reference this issue
on Oct 6, 2022 - added a commit that references this issue
on Jan 15, 2024 - added a commit that references this issue
on Feb 11, 2024
in module urllib.request， function getproxies_environment （https://github.com/python/cpython/blob/main/Lib/urllib/request.py#L2493）
iterate over all environment variables twice
I made a test:
so if there are many environment variables(like 10 thousands) in a machine, packages depend on function urllib.request.getproxies_environment (like requests does) shall be very slow
is 10 thousands environment variables reasonable？
yes it is, for example, in k8s cluster，kubelete may add a set of environment variables all Services(could be 10 thousands even 100 thousands), in purpose of service discovery
so would it be ok that getproxies_environment don't iterate over all environment variables?
Linked PRs