RuntimeError: Event loop is closed when AsyncClient exists in multiple event_loops.

[lee-cq]

I encountered a small problem. If I run the same AsyncClient in two event loops, an error will be reported in the second event loop (RuntimeError: Event loop is closed).

import asyncio

from httpx import AsyncClient

client = AsyncClient()

asyncio.run(client.get('https://www.baidu.com')).status_code
asyncio.run(client.get('https://www.baidu.com')).status_code

Error:

Traceback (most recent call last):
  File "/home/codespace/.local/lib/python3.10/site-packages/anyio/streams/tls.py", line 133, in _call_sslobject_method
    result = func(*args)
  File "/usr/local/python/3.10.8/lib/python3.10/ssl.py", line 917, in read
    v = self._sslobj.read(len)
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2548)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/codespace/.local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 111, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/home/codespace/.local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 176, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/home/codespace/.local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 212, in _receive_event
    data = await self._network_stream.read(
  File "/home/codespace/.local/lib/python3.10/site-packages/httpcore/_backends/anyio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/home/codespace/.local/lib/python3.10/site-packages/anyio/streams/tls.py", line 198, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/home/codespace/.local/lib/python3.10/site-packages/anyio/streams/tls.py", line 140, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/home/codespace/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1102, in receive
    self._transport.resume_reading()
  File "/usr/local/python/3.10.8/lib/python3.10/asyncio/selector_events.py", line 806, in resume_reading
    self._add_reader(self._sock_fd, self._read_ready)
  File "/usr/local/python/3.10.8/lib/python3.10/asyncio/selector_events.py", line 752, in _add_reader
    self._loop._add_reader(fd, callback, *args)
  File "/usr/local/python/3.10.8/lib/python3.10/asyncio/selector_events.py", line 253, in _add_reader
    self._check_closed()
  File "/usr/local/python/3.10.8/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspaces/alist-sdk/tmp/t1.py", line 8, in <module>
    asyncio.run(client.get('https://www.baidu.com')).status_code
  File "/usr/local/python/3.10.8/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/python/3.10.8/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/httpx/_client.py", line 1757, in get
    return await self.request(
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/codespace/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request
    raise exc
  File "/home/codespace/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/codespace/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/home/codespace/.local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 132, in handle_async_request
    await self._response_closed()
  File "/home/codespace/.local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 245, in _response_closed
    await self.aclose()
  File "/home/codespace/.local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 253, in aclose
    await self._network_stream.aclose()
  File "/home/codespace/.local/lib/python3.10/site-packages/httpcore/_backends/anyio.py", line 54, in aclose
    await self._stream.aclose()
  File "/home/codespace/.local/lib/python3.10/site-packages/anyio/streams/tls.py", line 195, in aclose
    await self.transport_stream.aclose()
  File "/home/codespace/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1161, in aclose
    self._transport.close()
  File "/usr/local/python/3.10.8/lib/python3.10/asyncio/selector_events.py", line 698, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/usr/local/python/3.10.8/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
    self._check_closed()
  File "/usr/local/python/3.10.8/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Can anyone tell me if this is normal?

[karpetrosyan]

Hi! This is because httpx uses connection pooling, and it will try to send the second request with the connection that it already has, but since asyncio cannot use the same stream in different event loops, you have an error.

To demonstrate it, you can send the special header in your http request, saying that you won't store the underlying connection in the connection pool, so you can make the second request in the separate event loop without any problem.

Doesn't work

asyncio.run(client.get('https://www.baidu.com')).status_code
asyncio.run(client.get('https://www.baidu.com')).status_code

Works

asyncio.run(client.get('https://www.baidu.com', headers=[('Connection', 'close')])).status_code
asyncio.run(client.get('https://www.baidu.com')).status_code

[tomchristie]

Well that's amusingly undefined.
We might get a connection close from the server even if we didn't request it.

Related... we probably ought to prefer just to always raise hard error for this case. (What behaviour do we see for trio here, or for the sync threading case?)

If anything it'd be neater if the async client only allowed async context block usage, so that we're forced to be explicit about it's scope. (Have we had any discussions about that, previously?)

[lee-cq]

Thank you.

[wssam-hassan]

I have faced the same issue,

I tried to use a lot of solutions, it didn't work.
when I used nest_asyncio package, it solves the issue.

Just put these 2 lines in the top of the script:
import nest_asyncio
nest_asyncio.apply()