Unexpected suppressing of asyncio.CancelledError exception

[decaz]

Environment:

• Python 3.10.7
• httpx 0.23.0

Listing:

import asyncio

import httpx


async def request(url):
    async with httpx.AsyncClient() as client:
        try:
            await client.get(url)
        except BaseException as exc:
            print('Request exception', type(exc), exc)
            raise


async def main():
    url = 'http://1.2.3.4:1234/'  # Some unreachable resource with delay
    task = asyncio.create_task(request(url))
    await asyncio.sleep(1)
    task.cancel()
    try:
        await task
    except BaseException as exc:
        print('Task exception', type(exc), exc)


asyncio.run(main())

Expected behavior:

Request exception <class 'asyncio.exceptions.CancelledError'> 
Task exception <class 'asyncio.exceptions.CancelledError'> 

Actual behavior:

Request exception <class 'httpx.ConnectError'> All connection attempts failed
Task exception <class 'httpx.ConnectError'> All connection attempts failed

[tomchristie]

I'm not able to replicate that...

import asyncio

import httpx


async def request(url):
    async with httpx.AsyncClient() as client:
        try:
            await client.get(url)
        except BaseException as exc:
            print('Request exception', type(exc), exc)
            raise


async def main():
    url = 'https://httpbin.org/delay/3'  # Some unreachable resource with delay
    task = asyncio.create_task(request(url))
    await asyncio.sleep(1)
    task.cancel()
    try:
        await task
    except BaseException as exc:
        print('Task exception', type(exc), exc)


asyncio.run(main())

Results in...

$ python ./example.py 
Request exception <class 'asyncio.exceptions.CancelledError'> 
Task exception <class 'asyncio.exceptions.CancelledError'> 

If I use an actually unreachable resource...

Eg. http://1.2.3.4:1234/, instead of https://httpbin.org/delay/3 then yes in that case I get a ConnectError.
Because the connect error occurs before cancellation is reached.

[decaz]

@tomchristie I have the following task: there is a coroutine where request should be done with guarantee, something like that:

async def request(url):
    while True:
        async with httpx.AsyncClient() as client:
            try:
                return await client.get(url)
            except httpx.RequestError as exc:
                await asyncio.sleep(5)

If I cancel a task with this coroutine on demand then it can suppress asyncio.CancelledError exception and continue to make requests so cancellation of this task will not be guaranteed - the task will hang.

[tomchristie]

It's not clear to me what you're asking there.

[decaz]

@tomchristie if you try to cancel this kind of task then httpx can raise httpx.ConnectError instead of asyncio.CancelledError and the task will hang because the loop won't be broken.

[decaz]

So the problem is there is no certainty which exception will be raised by httpx on cancellation of ongoing request.

[tomchristie]

If you'd like me to spend more time on this, could you share a reproducible example demonstrating the issue?

The example you've shared above gives a ConnectError for cases where it cannot connect, and a CancelledError for cases where it can connect but is then cancelled. I'm not seeing the issue there.

[decaz]

@tomchristie the example is similar to the initial listing:

import asyncio

import httpx


async def request(url):
    while True:
        async with httpx.AsyncClient() as client:
            print('Request: GET', url)
            try:
                return await client.get(url)
            except httpx.RequestError as exc:
                print('Request error, retrying in 5 sec', type(exc), exc)
                await asyncio.sleep(5)


async def main():
    url = 'http://1.2.3.4:1234/'
    print('Creating request task')
    task = asyncio.create_task(request(url))
    print('Sleeping for 5 sec')
    await asyncio.sleep(5)
    print('Cancelling request task')
    task.cancel()
    try:
        await task
    except BaseException as exc:
        print('Task exception', type(exc), exc)
    print('Request task cancelled')


asyncio.run(main())

This example demonstrates that "Request task cancelled" will never be printed so the main function (program) will hang forever:

Creating request task
Sleeping for 5 sec
Request: GET http://1.2.3.4:1234/
Cancelling request task
Request error, retrying in 5 sec <class 'httpx.ConnectError'> All connection attempts failed
Request: GET http://1.2.3.4:1234/
Request error, retrying in 5 sec <class 'httpx.ConnectTimeout'> 
Request: GET http://1.2.3.4:1234/
Request error, retrying in 5 sec <class 'httpx.ConnectTimeout'> 
Request: GET http://1.2.3.4:1234/
Request error, retrying in 5 sec <class 'httpx.ConnectTimeout'> 
Request: GET http://1.2.3.4:1234/
Request error, retrying in 5 sec <class 'httpx.ConnectTimeout'> 
Request: GET http://1.2.3.4:1234/
Request error, retrying in 5 sec <class 'httpx.ConnectTimeout'> 
...

[decaz]

I guess agronholm/anyio#374 is the reason for.

[decaz]

Fixed in agronholm/anyio#374.