Fixed Async `httpx` Request Handling in `vcr.py` Sync Contexts #833

fabianvf · 2024-04-03T18:01:28Z

Should now correctly handle asynchronous httpx requests when called from synchronous functions. Previously, using asyncio.run() in _sync_vcr_send caused issues in environments with an existing event loop, like aiohttp servers.

Solution:

Implemented a run_async_from_sync function using ThreadPoolExecutor to execute async tasks from synchronous functions.
Replaced asyncio.run() in _sync_vcr_send with run_async_from_sync.

This change ensures httpx requests are processed correctly in vcr.py's synchronous operations.

I think this is related to #817

Here's a reproducer:

import asyncio

import httpx
import pytest
from aiohttp import web

import vcr

my_vcr = vcr.VCR(cassette_library_dir='cassettes', record_mode='once')

@my_vcr.use_cassette('test.yml')
async def handle_request(request):
    client = httpx.Client()
    try:
        response = client.get('http://httpbin.org/get')
    finally:
        client.close()
    return web.Response(text=response.text, status=response.status_code)

def create_app():
    app = web.Application()
    app.router.add_get('/', handle_request)
    return app

@pytest.mark.asyncio
async def test_server(aiohttp_client):
    app = create_app()
    client = await aiohttp_client(app)
    resp = await client.get('/')
    assert resp.status == 200

(there's some issue on my branch with an unclosed event loop but I couldn't figure it out, I think it's related to the asyncio fixture but I'm not sure)

Before:

============================= test session starts ==============================
platform linux -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/fabian/projects/github.com/kevin1024/vcrpy
configfile: pyproject.toml
plugins: anyio-3.7.1, recording-0.13.1, asyncio-0.23.6, aiohttp-1.0.5
asyncio: mode=Mode.STRICT
collected 1 item

repro.py FE                                                              [100%]

==================================== ERRORS ====================================
_______________________ ERROR at teardown of test_server _______________________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_and_report.<locals>.<lambda> at 0x7fcb6d557c40>
when = 'teardown'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: Callable[[], TResult],
        when: Literal["collect", "setup", "call", "teardown"],
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.
    
        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

../../../../.local/lib/python3.12/site-packages/_pytest/runner.py:340: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../.local/lib/python3.12/site-packages/_pytest/runner.py:240: in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
../../../../.local/lib/python3.12/site-packages/pluggy/_hooks.py:501: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
../../../../.local/lib/python3.12/site-packages/pluggy/_manager.py:119: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
../../../../.local/lib/python3.12/site-packages/_pytest/threadexception.py:92: in pytest_runtest_teardown
    yield from thread_exception_runtest_hook()
../../../../.local/lib/python3.12/site-packages/_pytest/threadexception.py:63: in thread_exception_runtest_hook
    yield
../../../../.local/lib/python3.12/site-packages/_pytest/unraisableexception.py:95: in pytest_runtest_teardown
    yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            try:
                yield
            finally:
                if cm.unraisable:
                    if cm.unraisable.err_msg is not None:
                        err_msg = cm.unraisable.err_msg
                    else:
                        err_msg = "Exception ignored in"
                    msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                    msg += "".join(
                        traceback.format_exception(
                            cm.unraisable.exc_type,
                            cm.unraisable.exc_value,
                            cm.unraisable.exc_traceback,
                        )
                    )
>                   warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E                   pytest.PytestUnraisableExceptionWarning: Exception ignored in: <coroutine object _record_responses at 0x7fcb6d57bbc0>
E                   
E                   Traceback (most recent call last):
E                     File "/usr/lib64/python3.12/warnings.py", line 553, in _warn_unawaited_coroutine
E                       warn(msg, category=RuntimeWarning, stacklevel=2, source=coro)
E                   RuntimeWarning: coroutine '_record_responses' was never awaited

../../../../.local/lib/python3.12/site-packages/_pytest/unraisableexception.py:80: PytestUnraisableExceptionWarning
------------------------------ Captured log call -------------------------------
ERROR    aiohttp.server:web_protocol.py:421 Error handling request
Traceback (most recent call last):
  File "/home/fabian/.local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/aiohttp/web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/vcr/_handle_coroutine.py", line 3, in handle_coroutine
    return await fn(cassette)
           ^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/repro.py", line 15, in handle_request
    response = client.get('http://httpbin.org/get')
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 1055, in get
    return self.request(
           ^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 828, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 915, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 943, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 980, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/vcr/stubs/httpx_stubs.py", line 184, in _inner_send
    return _sync_vcr_send(cassette, real_send, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/vcr/stubs/httpx_stubs.py", line 177, in _sync_vcr_send
    asyncio.run(_record_responses(cassette, vcr_request, real_response, aread=False))
  File "/usr/lib64/python3.12/asyncio/runners.py", line 190, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop
=================================== FAILURES ===================================
_________________________________ test_server __________________________________

aiohttp_client = <function aiohttp_client.<locals>.go at 0x7fcb6d556f20>

    @pytest.mark.asyncio
    async def test_server(aiohttp_client):
        app = create_app()
        client = await aiohttp_client(app)
        resp = await client.get('/')
>       assert resp.status == 200
E       AssertionError: assert 500 == 200
E        +  where 500 = <ClientResponse(http://127.0.0.1:33117/) [500 Internal Server Error]>\n<CIMultiDictProxy('Content-Type': 'text/plain; c...Length': '55', 'Date': 'Wed, 03 Apr 2024 19:02:16 GMT', 'Server': 'Python/3.12 aiohttp/3.9.3', 'Connection': 'close')>\n.status

repro.py:32: AssertionError
------------------------------ Captured log call -------------------------------
ERROR    aiohttp.server:web_protocol.py:421 Error handling request
Traceback (most recent call last):
  File "/home/fabian/.local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/aiohttp/web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/vcr/_handle_coroutine.py", line 3, in handle_coroutine
    return await fn(cassette)
           ^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/repro.py", line 15, in handle_request
    response = client.get('http://httpbin.org/get')
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 1055, in get
    return self.request(
           ^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 828, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 915, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 943, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 980, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/vcr/stubs/httpx_stubs.py", line 184, in _inner_send
    return _sync_vcr_send(cassette, real_send, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/vcr/stubs/httpx_stubs.py", line 177, in _sync_vcr_send
    asyncio.run(_record_responses(cassette, vcr_request, real_response, aread=False))
  File "/usr/lib64/python3.12/asyncio/runners.py", line 190, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop
=========================== short test summary info ============================
FAILED repro.py::test_server - AssertionError: assert 500 == 200
ERROR repro.py::test_server - pytest.PytestUnraisableExceptionWarning: Except...
========================== 1 failed, 1 error in 0.80s ==========================

After:

============================= test session starts ==============================
platform linux -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/fabian/projects/github.com/kevin1024/vcrpy
configfile: pyproject.toml
plugins: anyio-3.7.1, recording-0.13.1, asyncio-0.23.6, aiohttp-1.0.5
asyncio: mode=Mode.STRICT
collected 1 item

repro.py .E                                                              [100%]

==================================== ERRORS ====================================
_______________________ ERROR at teardown of test_server _______________________

    def _close_event_loop() -> None:
        policy = asyncio.get_event_loop_policy()
        try:
            loop = policy.get_event_loop()
        except RuntimeError:
            loop = None
        if loop is not None:
            if not loop.is_closed():
>               warnings.warn(
                    _UNCLOSED_EVENT_LOOP_WARNING % loop,
                    DeprecationWarning,
                )
E               DeprecationWarning: pytest-asyncio detected an unclosed event loop when tearing down the event_loop
E               fixture: <_UnixSelectorEventLoop running=False closed=False debug=False>
E               pytest-asyncio will close the event loop for you, but future versions of the
E               library will no longer do so. In order to ensure compatibility with future
E               versions, please make sure that:
E                   1. Any custom "event_loop" fixture properly closes the loop after yielding it
E                   2. The scopes of your custom "event_loop" fixtures do not overlap
E                   3. Your code does not modify the event loop in async fixtures or tests

../../../../.local/lib/python3.12/site-packages/pytest_asyncio/plugin.py:818: DeprecationWarning
=========================== short test summary info ============================
ERROR repro.py::test_server - DeprecationWarning: pytest-asyncio detected an ...
========================== 1 passed, 1 error in 0.65s ==========================

Resolved an issue in `vcr.py`'s handling of `httpx` calls, where asynchronous `_record_responses` could not be directly called within a synchronous `_sync_vcr_send` function due to `asyncio.run()` limitations in already running event loops. Introduced a `run_async_from_sync` utility function using `ThreadPoolExecutor` to bridge this gap, allowing async functions to run effectively in sync contexts without event loop conflicts. This fix enables seamless handling of async `httpx` requests in `vcr.py`'s synchronous operations. - Added `run_async_from_sync` for async execution in sync contexts. - Replaced `asyncio.run()` with `run_async_from_sync` in `_sync_vcr_send`. Signed-off-by: Fabian von Feilitzsch <fabian@fabianism.us>

fabianvf · 2024-04-03T19:45:44Z

looks like this may be a dupe fix of #825

nilleb · 2024-04-27T19:40:06Z

looks like this may be a dupe fix of #825

Yes, maybe; I've tested both and yours just works (while the other don't).

Thanks, pal. 🙏🏻

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed Async `httpx` Request Handling in `vcr.py` Sync Contexts #833

Fixed Async `httpx` Request Handling in `vcr.py` Sync Contexts #833

fabianvf commented Apr 3, 2024 •

edited

fabianvf commented Apr 3, 2024

nilleb commented Apr 27, 2024

Fixed Async httpx Request Handling in vcr.py Sync Contexts #833

Are you sure you want to change the base?

Fixed Async httpx Request Handling in vcr.py Sync Contexts #833

Conversation

fabianvf commented Apr 3, 2024 • edited

fabianvf commented Apr 3, 2024

nilleb commented Apr 27, 2024

Fixed Async `httpx` Request Handling in `vcr.py` Sync Contexts #833

Fixed Async `httpx` Request Handling in `vcr.py` Sync Contexts #833

fabianvf commented Apr 3, 2024 •

edited