Respect system proxy exclusions.

[tomchristie]

• Initially raised as discussion httpx uses proxy unexpectedly on windows #1513

We're currently leaning on urllib.request.getproxies() to determine the system proxy setup, and setup which mounts should be a proxy transport and which should be a regular transport.

However, we're not using urllib.request.proxy_bypass(host).

This all works as expected when environment settings are being used. HTTP_PROXY, HTTPS_PROXY, and ALL_PROXY. In that case we're reading NO_PROXY, and ensuring anything hostname patterns there are mounted as a regular transport...

httpx/httpx/_utils.py

Lines 304 to 320 in 68cf1ff

	no_proxy_hosts = [host.strip() for host in proxy_info.get("no", "").split(",")]
	for hostname in no_proxy_hosts:
	# See https://curl.haxx.se/libcurl/c/CURLOPT_NOPROXY.html for details
	# on how names in `NO_PROXY` are handled.
	if hostname == "*":
	# If NO_PROXY=* is used or if "*" occurs as any one of the comma
	# seperated hostnames, then we should just bypass any information
	# from HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and always ignore
	# proxies.
	return {}
	elif hostname:
	# NO_PROXY=.google.com is marked as "all://*.google.com,
	# which disables "www.google.com" but not "google.com"
	# NO_PROXY=google.com is marked as "all://*google.com,
	# which disables "www.google.com" and "google.com".
	# (But not "wwwgoogle.com")
	mounts[f"all://*{hostname}"] = None

However, in the case when none of those environment variables are set getproxies() instead falls back to system proxy configuration. For windows this is registry based. ProxyEnable and ProxyOverride. For Mac this is sysconf based.
In those cases, we're correctly getting the configured proxies, but we aren't dealing with proxy exclusions.

We'd like to be able to setup these exclusions with our neat hostname pattern matched mounts system, which actually
means we can't just fallback to urllib.request.proxy_bypass(host), because that needs to be called per-host.

So, first steps...

• What exactly is the format of the windows registry ProxyOverride field?
• What exactly is the format of the "exceptions" field returned by from _scproxy import _get_proxy_settings()?

[tomchristie]

1. The windows registry ProxyOverride field.

A useful starting point is here...

https://github.com/python/cpython/blob/030a713183084594659aefd77b76fe30178e23c8/Lib/urllib/request.py#L2746

        proxyOverride = proxyOverride.split(';')
        # now check if we match one of the registry values.
        for test in proxyOverride:
            if test == '<local>':
                if '.' not in rawHost:
                    return 1
            test = test.replace(".", r"\.")     # mask dots
            test = test.replace("*", r".*")     # change glob sequence
            test = test.replace("?", r".")      # change glob char

2. The "exceptions" field returned by from _scproxy import _get_proxy_settings()

On my system this returns...

>>> _get_proxy_settings()
{'exclude_simple': False, 'exceptions': ('*.local', '169.254/16',)}

There's also an example documented in the urllib source code here...

https://github.com/python/cpython/blob/030a713183084594659aefd77b76fe30178e23c8/Lib/urllib/request.py#L2556

{ 'exclude_simple': bool, 'exceptions': ['foo.bar', '*.bar.com', '127.0.0.1', '10.1', '10.0/16']}

So, "10.0/16" and '169.254/16' here are not IPs, but IP ranges. Those are a bit awkward for us since we don't currently support subnet matching on transport mounts.

[amchii]

Thanks!

I think httpx can drop the support for system proxy settings, only uses environment settings:

1. It's hard to handle system proxy exclusions for httpx's mounts system.
2. If you use the system proxy settings, you should check the system proxy-bypass settings, otherwise it's not correct.

Besides, for the field 'exceptions' ,'10.1'='10.1/16'='10.1.1.1/16'.If httpx needs to mount system proxy-bypass settings on Windows and macosx, which can be unified into form like 10.1.* , then calls socket.gethostbyname to check both hostname and ip for request url, or simply not calls socket.gethostbyname which means DNS lookups is not supported.

[tomchristie]

Sticking to environment only settings would be one option, yes, though I'm not convinced that'd be the best from a user-experiance point of view.

[Jaharmi]

Being able to pick up and use the system proxy settings — especially proxy auto config — would be a benefit in certain environments, even if it’s an option and not the default. I’m no longer in a situation like that, but Python PAC support on macOS could have made configuring some projects much easier. It might have been a reason to choose a library like HTTPX over another.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tomchristie]

Still valid at the moment. Could do with a review and possibly extra docs.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tomchristie]

Upping the durations on you, @stalebot. Shoo.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[piankma]

Nothing's changed in this topic i presume?