httpx.ConnectError when performing many async requests

[olimcc]

We often experience httpx.ConnectErrors when making API calls with the httpx async client, we can reproduce this by passing many (~1000s) of async requests to the client. Oddly it manifests as a nodename nor servname provided exception. Based on some experimentation, it looks like this might be related to max_keepalive_connections and keepalive_expiry. To rule out other culprits (load balancer, service itself) I'm running an equivalent test with aiohttp, which has no problem completing the same volume of requests. It's a different exception, but it seems comparable to experiences noted in #1171. Is there a recommended configuration or approach I should use here?

I have found that passing 10,000 GETs to a pretty bare-bones AsyncClient, to a variety of hosts, from my local machine achieves it every time. I've shared a full testing script below. I'am typically able to achieve a successful outcome by tweaking httpx.Limits to some ~extreme settings:

# default setting results in failures
limits = httpx.Limits(max_connections=100, max_keepalive_connections=20, keepalive_expiry=5.0)
# these settings achieve success
limits = httpx.Limits(max_keepalive_connections=10000, keepalive_expiry=30)

In the case of error, the (abridged) exception:

ValueError: 'example.com' does not appear to be an IPv4 or IPv6 address
...
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
...
The above exception was the direct cause of the following exception:
httpcore.ConnectError: [Errno 8] nodename nor servname provided, or not known
...
The above exception was the direct cause of the following exception:
httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known

Full stack trace

Traceback (most recent call last):
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 192, in connect_tcp
    addr_obj = ip_address(remote_host)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/oli/.pyenv/versions/3.11.3/lib/python3.11/ipaddress.py", line 54, in ip_address
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
ValueError: 'example.com' does not appear to be an IPv4 or IPv6 address

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 195, in connect_tcp
    gai_res = await getaddrinfo(
              ^^^^^^^^^^^^^^^^^^
  File "/Users/oli/.pyenv/versions/3.11.3/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/oli/.pyenv/versions/3.11.3/lib/python3.11/socket.py", line 962, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 67, in map_httpcore_exceptions
    yield
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 371, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request
    raise exc
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
    raise exc
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
    stream = await self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpcore/_async/connection.py", line 124, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpcore/_backends/auto.py", line 30, in connect_tcp
    return await self._backend.connect_tcp(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 112, in connect_tcp
    with map_exceptions(exc_map):
  File "/Users/oli/.pyenv/versions/3.11.3/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 8] nodename nor servname provided, or not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/private/tmp/httpx-test/test.py", line 52, in <module>
    counter = asyncio.run(arun(method, host, nrequests))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/oli/.pyenv/versions/3.11.3/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/oli/.pyenv/versions/3.11.3/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/oli/.pyenv/versions/3.11.3/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/test.py", line 35, in arun
    statuses = await run_httpx(endpoint, nrequests)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/test.py", line 24, in run_httpx
    return [r.status_code for r in await asyncio.gather(*tasks)]
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/test.py", line 10, in get_url
    res = await client.get(endpoint)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpx/_client.py", line 1786, in get
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpx/_client.py", line 1559, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpx/_client.py", line 1646, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpx/_client.py", line 1674, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpx/_client.py", line 1711, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpx/_client.py", line 1748, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 370, in handle_async_request
    with map_httpcore_exceptions():
  File "/Users/oli/.pyenv/versions/3.11.3/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/private/tmp/httpx-test/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 84, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known

Testing script

import asyncio
import time
from collections import Counter

import aiohttp
import httpx

async def get_url(i, client, endpoint):
    print(f"GET {i}", endpoint)
    s = time.time()
    res = await client.get(endpoint)
    e = time.time()
    print(i, (e-s), res)
    return res


async def run_httpx(endpoint, nrequests):
    # the default httpx settings
    limits = httpx.Limits(max_connections=100, max_keepalive_connections=20, keepalive_expiry=5.0)
    # configuration that allows this task to succeed
    # limits = httpx.Limits(max_keepalive_connections=10000, keepalive_expiry=30)
    timeout = httpx.Timeout(None)
    async with httpx.AsyncClient(limits=limits, timeout=timeout) as client:
        tasks = [get_url(i, client, endpoint) for i in range(nrequests)]
        return [r.status_code for r in await asyncio.gather(*tasks)]

async def run_aiohttp(endpoint, nrequests):
    async with aiohttp.ClientSession() as session:
        tasks = [get_url(i, session, endpoint) for i in range(nrequests)]
        return [r.status for r in await asyncio.gather(*tasks)]

async def arun(method, endpoint, nrequests):
    c = Counter()
    if method == 'httpx':
        print('running httpx')
        statuses = await run_httpx(endpoint, nrequests)
    else:
        print('running aiohttp')
        statuses = await run_aiohttp(endpoint, nrequests)

    for s in statuses:
        c[s] += 1
    return c

if __name__ == '__main__':

    method = 'httpx'
    host = 'https://www.example.com'
    nrequests = 10_000
    assert method in ['aiohttp', 'httpx']
    s = time.time()
    counter = asyncio.run(arun(method, host, nrequests))
    print(counter.total(), counter)
    print(f'took {time.time()-s}')

Versions & configuration
Hardare: Mac / M1 Pro
Service target: uvicorn/fastapi, behind an AWS ALB (load balancer)
Python: 3.11.3

Pip freeze:

(venv) 09:37:43 /tmp/httpx-test$ pip freeze
aiohttp==3.9.3
aiosignal==1.3.1
anyio==4.2.0
attrs==23.2.0
certifi==2024.2.2
frozenlist==1.4.1
h11==0.14.0
httpcore==1.0.2
httpx==0.26.0
idna==3.6
multidict==6.0.5
sniffio==1.3.0
yarl==1.9.4