Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add httpx.Mounts transport class. #3070

Draft
wants to merge 9 commits into
base: master
Choose a base branch
from
Draft

Add httpx.Mounts transport class. #3070

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
d783502
Add Mounts transport class
tomchristie Jan 25, 2024
63da8c3
Cleanup
tomchristie Jan 25, 2024
216b93e
Cleanup
tomchristie Jan 25, 2024
3382fc9
Merge branch 'master' into mounts
tomchristie Jan 25, 2024
41f44fb
Update __init__.py
tomchristie Jan 25, 2024
1bf9305
Add tests
tomchristie Jan 31, 2024
dbc44f1
Tests and typing
tomchristie Jan 31, 2024
2812c10
Merge branch 'master' into mounts
tomchristie Jan 31, 2024
2d4cb7f
Merge branch 'master' into mounts
tomchristie Feb 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
12 changes: 5 additions & 7 deletions docs/advanced/proxies.md
View file
Open in desktop
Expand Up @@ -14,20 +14,18 @@ with httpx.Client(proxy="http://localhost:8030") as client:
...
```

For more advanced use cases, pass a mounts `dict`. For example, to route HTTP and HTTPS requests to 2 different proxies, respectively located at `http://localhost:8030`, and `http://localhost:8031`, pass a `dict` of proxy URLs:
For more advanced use cases, you might want to use the `httpx.Mounts` transport to determine which requests should be routed via a proxy:

```python
proxy_mounts = {
transport = httpx.Mounts({
"http://": httpx.HTTPTransport(proxy="http://localhost:8030"),
"https://": httpx.HTTPTransport(proxy="http://localhost:8031"),
}
"https://": httpx.HTTPTransport(proxy="https://localhost:8031"),
})

with httpx.Client(mounts=proxy_mounts) as client:
with httpx.Client(transport=transport) as client:
...
```

For detailed information about proxy routing, see the [Routing](#routing) section.

!!! tip "Gotcha"
In most cases, the proxy URL for the `https://` key _should_ use the `http://` scheme (that's not a typo!).

Expand Down
289 changes: 120 additions & 169 deletions docs/advanced/transports.md
View file
Open in desktop
Expand Up @@ -78,50 +78,6 @@ with httpx.Client(transport=transport, base_url="http://testserver") as client:
...
```

## Custom transports

A transport instance must implement the low-level Transport API, which deals
with sending a single request, and returning a response. You should either
subclass `httpx.BaseTransport` to implement a transport to use with `Client`,
or subclass `httpx.AsyncBaseTransport` to implement a transport to
use with `AsyncClient`.

At the layer of the transport API we're using the familiar `Request` and
`Response` models.

See the `handle_request` and `handle_async_request` docstrings for more details
on the specifics of the Transport API.

A complete example of a custom transport implementation would be:

```python
import json
import httpx


class HelloWorldTransport(httpx.BaseTransport):
"""
A mock transport that always returns a JSON "Hello, world!" response.
"""

def handle_request(self, request):
message = {"text": "Hello, world!"}
content = json.dumps(message).encode("utf-8")
stream = httpx.ByteStream(content)
headers = [(b"content-type", b"application/json")]
return httpx.Response(200, headers=headers, stream=stream)
```

Which we can use in the same way:

```pycon
>>> import httpx
>>> client = httpx.Client(transport=HelloWorldTransport())
>>> response = client.get("https://example.org/")
>>> response.json()
{"text": "Hello, world!"}
```

## Mock transports

During testing it can often be useful to be able to mock out a transport,
Expand Down Expand Up @@ -149,184 +105,179 @@ mocking library, RESPX](https://lundberg.github.io/respx/), or the [pytest-httpx

## Mounting transports

You can also mount transports against given schemes or domains, to control
which transport an outgoing request should be routed via, with [the same style
used for specifying proxy routing](#routing).
The `httpx.Mounts` and `httpx.AsyncMounts` transport classes allow you to map URL patterns to HTTP transports. HTTPX matches requested URLs against URL patterns to decide which transport should be used, if any. Matching is done from most specific URL patterns (e.g. `https://<domain>:<port>`) to least specific ones (e.g. `https://`).

```python
import httpx
HTTPX supports routing requests based on **scheme**, **domain**, **port**, or a combination of these.

class HTTPSRedirectTransport(httpx.BaseTransport):
"""
A transport that always redirects to HTTPS.
"""
Make sure that you're using `httpx.Mounts` if you're working with `httpx.Client` instance, or `httpx.AsyncMounts` if you're working with `httpx.AsyncClient`.

def handle_request(self, method, url, headers, stream, extensions):
scheme, host, port, path = url
if port is None:
location = b"https://%s%s" % (host, path)
else:
location = b"https://%s:%d%s" % (host, port, path)
stream = httpx.ByteStream(b"")
headers = [(b"location", location)]
extensions = {}
return 303, headers, stream, extensions
**Scheme routing**

URL patterns should start with a scheme.

# A client where any `http` requests are always redirected to `https`
mounts = {'http://': HTTPSRedirectTransport()}
client = httpx.Client(mounts=mounts)
The following example routes HTTP requests through a proxy, and send HTTPS requests directly...

```python
transport = httpx.Mounts({
"http://": httpx.HTTPTransport(proxy="http://localhost:8030"),
"https://": httpx.HTTPTransport(),
})
client = httpx.Client(transport=transport)
```

A couple of other sketches of how you might take advantage of mounted transports...
**Domain routing**

The `all://` scheme will match any URL scheme. URL patterns may also include a domain.

Disabling HTTP/2 on a single given domain...
Uses cases here might include setting particular SSL or HTTP version configurations for a given domain. For example, disabling HTTP/2 on one particular site...

```python
mounts = {
transport = httpx.Mounts({
"all://": httpx.HTTPTransport(http2=True),
"all://*example.org": httpx.HTTPTransport()
}
client = httpx.Client(mounts=mounts)
})
client = httpx.Client(transport=transport)
```

Mocking requests to a given domain:
An alternate use case might be sending all requests on a given domain to a mock application, while allowing all other requests to pass through...

```python
# All requests to "example.org" should be mocked out.
# Other requests occur as usual.
def handler(request):
return httpx.Response(200, json={"text": "Hello, World!"})

mounts = {"all://example.org": httpx.MockTransport(handler)}
client = httpx.Client(mounts=mounts)
```
import httpx

Adding support for custom schemes:
def hello_world(request: httpx.Request):
return httpx.Response("<p>Hello, World!</p>")

```python
# Support URLs like "file:///Users/sylvia_green/websites/new_client/index.html"
mounts = {"file://": FileSystemTransport()}
client = httpx.Client(mounts=mounts)
transport = httpx.Mounts({
"all://example.com": httpx.MockTransport(app=hello_world),
"all://": httpx.HTTPTransport()
})
client = httpx.Client(transport=transport)
```

### Routing

HTTPX provides a powerful mechanism for routing requests, allowing you to write complex rules that specify which transport should be used for each request.

The `mounts` dictionary maps URL patterns to HTTP transports. HTTPX matches requested URLs against URL patterns to decide which transport should be used, if any. Matching is done from most specific URL patterns (e.g. `https://<domain>:<port>`) to least specific ones (e.g. `https://`).

HTTPX supports routing requests based on **scheme**, **domain**, **port**, or a combination of these.

### Wildcard routing

Route everything through a transport...
If we would like to also include subdomains we can use a leading wildcard...

```python
mounts = {
"all://": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
transport = httpx.Mounts({
"all://*example.com": httpx.MockTransport(app=hello_world),
"all://": httpx.HTTPTransport()
})
```

### Scheme routing

Route HTTP requests through one transport, and HTTPS requests through another...
Or if we want to only route to *subdomains* of `example.com`, but allow all other requests to pass through...

```python
mounts = {
"http://": httpx.HTTPTransport(proxy="http://localhost:8030"),
"https://": httpx.HTTPTransport(proxy="http://localhost:8031"),
}
transport = httpx.Mounts({
"all://*.example.com": httpx.MockTransport(app=example_subdomains),
"all://example.com": httpx.MockTransport(app=example),
"all://": httpx.HTTPTransport(),
})
```

### Domain routing
**Port routing**

Proxy all requests on domain "example.com", let other requests pass through...
The URL patterns also support port matching...

```python
mounts = {
"all://example.com": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
transport = httpx.Mounts({
"all://*:1234": httpx.HTTPTransport(proxy="http://localhost:8030"),
"all://": httpx.HTTPTransport(),
})
client = httpx.Client(transport=transport)
```

Proxy HTTP requests on domain "example.com", let HTTPS and other requests pass through...
## Custom transports

```python
mounts = {
"http://example.com": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```
A transport instance must implement the low-level Transport API which deals
with sending a single request, and returning a response. You should either
subclass `httpx.BaseTransport` to implement a transport to use with `Client`,
or subclass `httpx.AsyncBaseTransport` to implement a transport to
use with `AsyncClient`.

Proxy all requests to "example.com" and its subdomains, let other requests pass through...
At the layer of the transport API we're using the familiar `Request` and
`Response` models.

```python
mounts = {
"all://*example.com": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```
See the `handle_request` and `handle_async_request` docstrings for more details
on the specifics of the Transport API.

Proxy all requests to strict subdomains of "example.com", let "example.com" and other requests pass through...
A complete example of a custom transport implementation would be:

```python
mounts = {
"all://*.example.com": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

### Port routing
import json
import httpx

Proxy HTTPS requests on port 1234 to "example.com"...
class HelloWorldTransport(httpx.BaseTransport):
"""
A mock transport that always returns a JSON "Hello, world!" response.
"""

```python
mounts = {
"https://example.com:1234": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
def handle_request(self, request):
return httpx.Response(200, json={"text": "Hello, world!"})
```

Proxy all requests on port 1234...
Or this example, which uses a custom transport and `httpx.Mounts` to always redirect `http://` requests.

```python
mounts = {
"all://*:1234": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

### No-proxy support
class HTTPSRedirect(httpx.BaseTransport):
"""
A transport that always redirects to HTTPS.
"""
def handle_request(self, request):
url = request.url.copy_with(scheme="https")
return httpx.Response(303, headers={"Location": str(url)})

It is also possible to define requests that _shouldn't_ be routed through the transport.
# A client where any `http` requests are always redirected to `https`
transport = httpx.Mounts({
'http://': HTTPSRedirect()
'https://': httpx.HTTPTransport()
})
client = httpx.Client(transport=transport)
```

To do so, pass `None` as the proxy URL. For example...
A useful pattern here is custom transport classes that wrap the default HTTP implementation. For example...

```python
mounts = {
# Route requests through a proxy by default...
"all://": httpx.HTTPTransport(proxy="http://localhost:8031"),
# Except those for "example.com".
"all://example.com": None,
}
```
class DebuggingTransport(httpx.BaseTransport):
def __init__(self, **kwargs):
self._wrapper = httpx.HTTPTransport(**kwargs)

### Complex configuration example
def handle_request(self, request):
print(f">>> {request}")
response = self._wrapper.handle_request(request)
print(f"<<< {response}")
return response

You can combine the routing features outlined above to build complex proxy routing configurations. For example...
def close(self):
self._wrapper.close()

```python
mounts = {
# Route all traffic through a proxy by default...
"all://": httpx.HTTPTransport(proxy="http://localhost:8030"),
# But don't use proxies for HTTPS requests to "domain.io"...
"https://domain.io": None,
# And use another proxy for requests to "example.com" and its subdomains...
"all://*example.com": httpx.HTTPTransport(proxy="http://localhost:8031"),
# And yet another proxy if HTTP is used,
# and the "internal" subdomain on port 5550 is requested...
"http://internal.example.com:5550": httpx.HTTPTransport(proxy="http://localhost:8032"),
}
transport = DebuggingTransport()
client = httpx.Client(transport=transport)
```

### Environment variables
Here's another case, where we're using a round-robin across a number of different proxies...

There are also environment variables that can be used to control the dictionary of the client mounts.
They can be used to configure HTTP proxying for clients.
```python
class ProxyRoundRobin(httpx.BaseTransport):
def __init__(self, proxies, **kwargs):
self._transports = [
httpx.HTTPTransport(proxy=proxy, **kwargs)
for proxy in proxies
]
self._idx = 0

See documentation on [`HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`](../environment_variables.md#http_proxy-https_proxy-all_proxy) for more information.
def handle_request(self, request):
transport = self._transports[self._idx]
self._idx = (self._idx + 1) % len(self._transports)
return transport.handle_request(request)

def close(self):
for transport in self._transports:
transport.close()

proxies = [
httpx.Proxy("http://127.0.0.1:8081"),
httpx.Proxy("http://127.0.0.1:8082"),
httpx.Proxy("http://127.0.0.1:8083"),
]
transport = ProxyRoundRobin(proxies=proxies)
client = httpx.Client(transport=transport)
```